Nov 03 16:09:53.045507 ip-10-0-135-216 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:09:53.474895 ip-10-0-135-216 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:09:53.474895 ip-10-0-135-216 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:09:53.474895 ip-10-0-135-216 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:09:53.474895 ip-10-0-135-216 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:09:53.480140 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480042 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.480140 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480135 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.480140 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480140 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.480140 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480144 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.480140 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480148 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.480140 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480152 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480156 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480160 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480164 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480167 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480171 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480174 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480178 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480181 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480184 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480187 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480191 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480195 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480199 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480222 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480229 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480233 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480237 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480241 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.480512 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480244 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480248 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480258 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480264 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480271 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480275 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480279 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480283 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480287 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480291 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480294 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480298 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480303 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480307 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480310 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480316 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480320 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480325 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480329 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480334 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.481286 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480338 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480342 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480347 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480351 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480356 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480360 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480364 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480369 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480375 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480379 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480383 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480388 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480392 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480396 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480401 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480405 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480409 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480413 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480420 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.482100 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480425 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480429 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480434 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480438 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480443 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480447 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480451 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480455 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480460 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480465 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480469 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480473 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480477 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480481 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480486 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480490 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480494 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480498 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480502 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480506 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.482894 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480512 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480516 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480520 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480524 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480527 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480532 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480536 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480540 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480547 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480551 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480556 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480560 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480564 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480569 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480573 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480577 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480581 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480585 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480590 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.483740 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480593 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480599 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480604 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.480608 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481958 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481968 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481975 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481980 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481985 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481990 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481994 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.481998 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482003 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482007 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482012 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482016 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482020 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482025 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482030 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.484553 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482034 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482038 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482043 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482047 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482051 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482056 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482060 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482064 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482068 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482072 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482076 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482081 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482086 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482090 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482095 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482099 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482103 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482108 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482112 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482116 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.485392 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482120 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482124 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482129 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482133 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482137 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482142 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482146 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482150 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482156 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482160 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482164 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482169 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482174 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482178 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482182 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482186 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482190 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482195 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482199 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.486277 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482221 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482226 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482231 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482235 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482239 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482243 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482247 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482253 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482257 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482261 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482266 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482271 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482275 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482282 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482289 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482296 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482300 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482305 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482309 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.486778 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482313 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482317 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482321 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482328 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482333 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482337 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482341 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482345 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482351 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482355 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482359 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482363 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482367 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482371 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482375 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482379 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482384 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482388 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482392 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482396 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.487565 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482402 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482406 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482411 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482415 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482419 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482424 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482428 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482432 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482436 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482440 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482445 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482449 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.482453 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483411 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483422 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483429 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483433 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483438 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483442 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483447 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.488031 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483451 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483455 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483459 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483464 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483468 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483472 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483477 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483481 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483485 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483489 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483493 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483499 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483504 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483509 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483513 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483517 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483521 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483525 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483530 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.488549 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483534 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483538 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483542 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483547 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483551 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483555 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483560 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483564 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483569 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483573 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483578 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483582 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483586 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483590 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483594 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483598 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483602 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483606 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483610 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483614 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.489053 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483618 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483622 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483627 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483631 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483638 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483646 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483650 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483654 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483658 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483663 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483667 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483671 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483675 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483679 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483683 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483688 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483692 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483696 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483700 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.489576 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483705 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483712 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483719 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483725 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483731 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483736 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483741 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483745 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483749 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483754 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483758 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483763 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483767 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483771 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483775 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483780 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483784 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483791 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483796 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.490046 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483801 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483805 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483809 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483813 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483817 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483821 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483825 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483830 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483834 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483840 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483847 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483851 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483856 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483860 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483864 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483869 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483873 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483878 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483885 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.490523 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483889 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483893 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.483897 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484034 2572 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484046 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484056 2572 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484063 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484070 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484076 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484082 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484089 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484097 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484103 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484109 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484114 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484119 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484124 2572 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484128 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484133 2572 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484138 2572 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484143 2572 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484149 2572 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484154 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484159 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:09:53.490997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484164 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484170 2572 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484176 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484181 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484186 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484191 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484195 2572 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484224 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484230 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484235 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484240 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484247 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484252 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484256 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484261 2572 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484265 2572 flags.go:64] FLAG: --enable-server="true" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484270 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484279 2572 flags.go:64] FLAG: --event-burst="100" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484285 2572 flags.go:64] FLAG: --event-qps="50" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484291 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484298 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484302 2572 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484309 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484314 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484319 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:09:53.491571 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484324 2572 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484328 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484333 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484338 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484343 2572 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484347 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484352 2572 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484356 2572 flags.go:64] FLAG: --feature-gates="" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484367 2572 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484372 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484377 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484382 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484386 2572 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484391 2572 flags.go:64] FLAG: --help="false" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484396 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484402 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484407 2572 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484412 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484418 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484423 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484428 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484432 2572 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484436 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:09:53.492297 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484441 2572 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484446 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484451 2572 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484457 2572 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484463 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484468 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484472 2572 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484477 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484482 2572 flags.go:64] FLAG: --lock-file="" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484487 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484491 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484496 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484506 2572 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484511 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484517 2572 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484522 2572 flags.go:64] FLAG: --logging-format="text" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484527 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484532 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484537 2572 flags.go:64] FLAG: --manifest-url="" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484541 2572 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484548 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484553 2572 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484559 2572 flags.go:64] FLAG: --max-pods="110" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484563 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484568 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:09:53.492849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484575 2572 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484580 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484585 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484589 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484594 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484606 2572 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484611 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484616 2572 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484621 2572 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484625 2572 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484631 2572 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484638 2572 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484642 2572 flags.go:64] FLAG: --port="10250" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484647 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484652 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-05e91847f7ec0d696" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484657 2572 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484662 2572 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484667 2572 flags.go:64] FLAG: --register-node="true" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484671 2572 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484678 2572 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484683 2572 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484687 2572 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484692 2572 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484698 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484703 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484707 2572 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:09:53.493462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484712 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484717 2572 flags.go:64] FLAG: --runonce="false" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484722 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484727 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484732 2572 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484737 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484742 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484747 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484752 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484757 2572 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484762 2572 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484767 2572 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484771 2572 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484776 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484781 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484785 2572 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484791 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484802 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484806 2572 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484811 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484817 2572 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484822 2572 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484826 2572 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484831 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484836 2572 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:09:53.494085 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484841 2572 flags.go:64] FLAG: --v="2" Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484848 2572 flags.go:64] FLAG: --version="false" Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484855 2572 flags.go:64] FLAG: --vmodule="" Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484861 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.484866 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485116 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485125 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485130 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485135 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485139 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485143 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485147 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485151 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485156 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485160 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485165 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485169 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485174 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485178 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485182 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.494688 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485187 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485191 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485195 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485219 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485226 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485230 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485235 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485239 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485243 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485248 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485252 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485256 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485261 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485265 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485269 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485273 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485277 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485283 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485287 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485291 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.495162 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485295 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485300 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485304 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485308 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485312 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485316 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485320 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485327 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485335 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485340 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485344 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485348 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485353 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485357 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485361 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485367 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485373 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485377 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485381 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.495692 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485385 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485389 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485393 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485397 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485401 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485405 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485409 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485413 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485418 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485422 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485426 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485430 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485435 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485439 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485443 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485448 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485452 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485457 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485461 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485465 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.496138 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485470 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485474 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485479 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485483 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485487 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485492 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485496 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485500 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485505 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485509 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485514 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485518 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485522 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485527 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485531 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485535 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485539 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485544 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485548 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.496642 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485554 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485559 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485565 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485570 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485575 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485579 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485584 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485588 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485592 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485596 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485600 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485604 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485608 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.485653 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485724 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485732 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.497123 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485738 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485742 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485746 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485750 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485754 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485759 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485763 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485768 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485772 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485776 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485780 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485784 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485789 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485793 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485797 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485801 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485805 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485810 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485814 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485818 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.497525 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485822 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485827 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485831 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485835 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485839 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485842 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485846 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485851 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485855 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485859 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485864 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485868 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485872 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485876 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485880 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485885 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485889 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485895 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485902 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.498006 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485906 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485911 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485915 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485919 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485923 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485927 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485931 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485935 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485939 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485944 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485948 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485952 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485956 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485960 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485964 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485968 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485972 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485976 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485980 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.498460 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485984 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485989 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485993 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.485997 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486001 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486005 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486009 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486013 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486017 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486022 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486026 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486030 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486034 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486040 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486044 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486048 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486052 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486056 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486060 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486064 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.499008 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486068 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486072 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486076 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486080 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486085 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486089 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486093 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486097 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486101 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486105 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486109 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486113 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486117 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486121 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486127 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486134 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486138 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486142 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486146 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.499506 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486152 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486156 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486161 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486165 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486169 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486177 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.486182 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.493673 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.493691 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.493705 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.493710 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493767 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493772 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493775 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493778 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493782 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493784 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493787 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493789 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.499968 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493792 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493794 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493797 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493800 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493802 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493805 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493807 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493810 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493813 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493815 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493818 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493820 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493823 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493825 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493829 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493833 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493837 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493839 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493842 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493845 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.500550 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493848 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493851 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493854 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493857 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493859 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493861 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493864 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493866 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493869 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493871 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493874 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493876 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493879 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493881 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493883 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493886 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493888 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493891 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493894 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493896 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.501030 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493898 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493901 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493903 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493906 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493908 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493910 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493912 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493915 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493917 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493919 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493922 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493924 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493927 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493929 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493933 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493935 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493938 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493940 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493943 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.501561 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493945 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493948 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493950 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493953 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493955 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493957 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493960 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493962 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493965 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493967 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493970 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493972 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493974 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493977 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493979 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493982 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493985 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493987 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493990 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.502073 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493992 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493995 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.493997 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494000 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494002 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494005 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494007 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494010 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494014 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494018 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494022 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494024 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494027 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494029 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494032 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494034 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494037 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494039 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494042 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.502541 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494044 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.494071 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494223 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494228 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494231 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494234 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494237 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494240 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494242 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494244 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494247 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494250 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494253 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494255 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494258 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494260 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.503243 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494262 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494265 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494267 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494270 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494273 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494275 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494278 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494281 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494284 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494286 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494288 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494291 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494293 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494296 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494298 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494301 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494303 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494305 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494308 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.503919 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494310 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494312 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494315 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494317 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494319 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494323 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494326 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494329 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494332 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494335 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494339 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494342 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494346 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494348 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494351 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494354 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494356 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494359 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494361 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.504794 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494364 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494366 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494369 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494372 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494374 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494376 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494379 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494381 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494384 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494386 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494388 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494391 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494394 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494396 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494398 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494401 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494403 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494406 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494409 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494411 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.505335 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494414 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494416 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494419 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494421 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494424 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494426 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494428 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494431 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494437 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494439 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494442 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494444 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494447 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494449 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494452 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494455 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494457 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494460 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494462 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494465 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.505848 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494467 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494470 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494472 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494475 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494477 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494480 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494482 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494485 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494487 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494489 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494492 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494494 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494497 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494499 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.494522 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494553 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:09:53.506414 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494558 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494561 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494565 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494569 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494571 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494576 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494578 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494581 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494583 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494586 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494588 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494591 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494594 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494596 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494599 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494601 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494604 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494606 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494609 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:09:53.506784 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494611 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494614 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494616 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494619 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494621 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494624 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494626 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494629 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494631 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494633 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494636 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494639 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494641 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494643 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494646 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494648 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494650 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494653 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494657 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494659 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:09:53.507278 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494661 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494664 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494666 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494669 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494671 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494674 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494677 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494679 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494681 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494684 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494686 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494689 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494692 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494694 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494697 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494700 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494702 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494705 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494707 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494710 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:09:53.507742 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494713 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494715 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494718 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494721 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494723 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494725 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494728 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494730 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494732 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494735 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494738 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494741 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494743 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494746 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494748 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494751 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494753 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494756 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494758 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:09:53.508298 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494767 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494770 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494772 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494775 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494777 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494780 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494783 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494785 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494788 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494790 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494792 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494795 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494797 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494800 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494802 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494805 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494808 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494811 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494813 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:09:53.508741 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494816 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494818 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494821 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494823 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494826 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494829 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494833 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:53.494837 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.494895 2572 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.496862 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.498446 2572 server.go:1014] "Starting client certificate rotation" Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.498550 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.498585 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.501493 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:09:53.509307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.505992 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:09:53.519045 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.519023 2572 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:09:53.522455 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.522439 2572 log.go:25] "Validated CRI v1 image API" Nov 03 16:09:53.523248 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.523220 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:09:53.528741 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.528721 2572 fs.go:135] Filesystem UUIDs: map[5980846a-5135-4947-b119-a271f82233d5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7aab4194-4be6-4a56-9482-f2648c83cfbc:/dev/nvme0n1p3] Nov 03 16:09:53.528787 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.528742 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 16:09:53.532957 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.532849 2572 manager.go:217] Machine: {Timestamp:2026-11-03 16:09:53.532399016 +0000 UTC m=+0.360509383 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079832 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24928d62b2703c26b8aa9347942485 SystemUUID:ec24928d-62b2-703c-26b8-aa9347942485 BootID:a46a415a-0dac-4aa4-a86b-3668d7bf5e5a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ee:8e:d4:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ee:8e:d4:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:b9:56:9b:84:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:09:53.532957 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.532956 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:09:53.533066 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.533055 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:09:53.536465 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.536440 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:09:53.536594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.536466 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-216.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:09:53.536640 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.536597 2572 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:09:53.536640 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.536605 2572 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:09:53.536640 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.536615 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:09:53.537403 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.537393 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:09:53.538356 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.538345 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:09:53.541948 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.541936 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:09:53.542061 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.542051 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:09:53.542722 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.542713 2572 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:09:53.542752 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.542727 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:09:53.542752 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.542738 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:09:53.542752 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.542749 2572 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:09:53.542826 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.542758 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:09:53.543043 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.543031 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:09:53.545049 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.545032 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:09:53.549800 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.549787 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:09:53.550587 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550575 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:09:53.550632 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550605 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:09:53.550762 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550749 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:09:53.550800 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550764 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:09:53.550800 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550771 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:09:53.550800 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550776 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:09:53.550800 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550792 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:09:53.550800 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.550798 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:09:53.551405 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.551396 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:09:53.551436 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.551406 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:09:53.551436 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.551414 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:09:53.551436 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.551419 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:09:53.551436 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.551433 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:09:53.551552 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.551442 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:09:53.552877 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.552868 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:09:53.552914 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.552879 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:09:53.556732 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.556719 2572 server.go:1263] "Started kubelet" Nov 03 16:09:53.556808 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.556788 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:09:53.560721 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.556874 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:09:53.560926 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.560853 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:09:53.561879 ip-10-0-135-216 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:09:53.563326 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.563185 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:09:53.564125 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.564107 2572 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:09:53.567615 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.567593 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 16:09:53.568354 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.568340 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:09:53.568399 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.568341 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:09:53.568459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.568439 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:53.569102 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569081 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:09:53.569224 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569193 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:09:53.569224 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569221 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:09:53.569494 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569225 2572 factory.go:55] Registering systemd factory Nov 03 16:09:53.569494 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569243 2572 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:09:53.569494 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569331 2572 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:09:53.569494 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569337 2572 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:09:53.569678 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569571 2572 factory.go:153] Registering CRI-O factory Nov 03 16:09:53.569678 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569585 2572 factory.go:223] Registration of the crio container factory successfully Nov 03 16:09:53.569678 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569641 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:09:53.569678 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569664 2572 factory.go:103] Registering Raw factory Nov 03 16:09:53.569864 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.569687 2572 manager.go:1199] Started watching for new ooms in manager Nov 03 16:09:53.570733 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.570716 2572 manager.go:319] Starting recovery of all containers Nov 03 16:09:53.582352 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.582331 2572 manager.go:324] Recovery completed Nov 03 16:09:53.587551 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.587537 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:53.592049 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.592028 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:53.592130 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.592063 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:53.592130 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.592088 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:53.592754 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.592737 2572 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:09:53.592754 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.592753 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:09:53.593318 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.593299 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:53.593773 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.593749 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:09:53.594608 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.594588 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:53.594707 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.594650 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:53.594707 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.594666 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:53.596632 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.596619 2572 policy_none.go:50] "Start" Nov 03 16:09:53.596632 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.596635 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:09:53.596728 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.596646 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:09:53.598644 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.598631 2572 policy_none.go:44] "Start" Nov 03 16:09:53.645849 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.645815 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:09:53.648197 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.648175 2572 manager.go:341] "Starting Device Plugin manager" Nov 03 16:09:53.648315 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.648242 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:09:53.648315 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.648254 2572 server.go:87] "Starting device plugin registration server" Nov 03 16:09:53.648548 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.648535 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:09:53.648587 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.648550 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:09:53.648681 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.648643 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:09:53.648807 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.648766 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:09:53.648807 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.648777 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:09:53.649333 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.649311 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:09:53.649396 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.649371 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:53.690533 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.690509 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-216.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 16:09:53.692097 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.690509 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-216.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 16:09:53.692097 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.690504 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-216.us-west-2.compute.internal.18e494d4dcae5323 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-216.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-216.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 16:09:53.556689699 +0000 UTC m=+0.384800066,LastTimestamp:2026-11-03 16:09:53.556689699 +0000 UTC m=+0.384800066,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-216.us-west-2.compute.internal,}" Nov 03 16:09:53.706581 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.706553 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:09:53.707917 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.707902 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:09:53.708003 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.707924 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:09:53.708003 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.707944 2572 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:09:53.708003 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.707981 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 16:09:53.742045 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.741966 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-216.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 16:09:53.742124 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.742005 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-216.us-west-2.compute.internal.18e494d4dec9dce3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-216.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-216.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-216.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 16:09:53.592048867 +0000 UTC m=+0.420159234,LastTimestamp:2026-11-03 16:09:53.592048867 +0000 UTC m=+0.420159234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-216.us-west-2.compute.internal,}" Nov 03 16:09:53.749252 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.749237 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:53.750009 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.749994 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:53.750063 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.750020 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:53.750063 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.750034 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:53.750063 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.750057 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.792911 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.792890 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:09:53.792993 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.792953 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6mg2l" Nov 03 16:09:53.808665 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.808629 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal"] Nov 03 16:09:53.808763 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.808703 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:53.809505 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.809491 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:53.809580 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.809522 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:53.809580 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.809544 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:53.811572 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.811557 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:53.811719 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.811705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.811770 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.811761 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:53.812345 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.812329 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:53.812393 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.812362 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:53.812393 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.812370 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:53.812393 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.812377 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:53.812393 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.812388 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:53.812512 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.812400 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:53.814321 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.814309 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.814360 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.814333 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:53.815001 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.814986 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:53.815068 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.815017 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:53.815068 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.815027 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:53.843244 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.843220 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" node="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.849601 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.849579 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" node="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.850316 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.850301 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:09:53.870641 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.870617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4357665f1982cbd14d16f18df231ee45-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"4357665f1982cbd14d16f18df231ee45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.870731 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.870644 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.870731 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.870663 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.952585 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.952554 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:09:53.955697 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.955437 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.955697 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:53.955469 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-216.us-west-2.compute.internal\": node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:53.955880 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.955798 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:09:53.971548 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.971521 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.971669 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.971550 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4357665f1982cbd14d16f18df231ee45-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"4357665f1982cbd14d16f18df231ee45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.971669 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.971567 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.971669 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.971621 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.971669 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.971639 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4357665f1982cbd14d16f18df231ee45-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"4357665f1982cbd14d16f18df231ee45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:53.971669 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:53.971640 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:54.077317 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.077240 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.147050 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.147018 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:54.152817 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.152800 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:54.177333 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.177306 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.278304 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.278275 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.378930 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.378851 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.479284 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.479241 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.497423 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.497397 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 16:09:54.497590 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.497555 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:09:54.497706 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.497592 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:09:54.497706 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.497600 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:09:54.497706 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.497607 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7ccfbe3a02a843d98ab5cd19bd85dc7-f84d5194377c4192.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-216.us-west-2.compute.internal.18e494d4deca8616\": read tcp 10.0.135.216:45950->3.22.210.173:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-216.us-west-2.compute.internal.18e494d4deca8616 default 5416 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-216.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-216.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-216.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 16:09:53 +0000 UTC,LastTimestamp:2026-11-03 16:09:53.750037784 +0000 UTC m=+0.578148155,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-216.us-west-2.compute.internal,}" Nov 03 16:09:54.580272 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.580236 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.680915 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.680852 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.744588 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.744562 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:09:54.745447 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.745429 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:09:54.745502 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.745462 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:09:54.745502 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.745473 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:09:54.781569 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.781543 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.797216 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.797178 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:04:54 +0000 UTC" deadline="2028-07-15 22:13:00.181854256 +0000 UTC" Nov 03 16:09:54.797268 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.797216 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14886h3m5.384653389s" Nov 03 16:09:54.882366 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:54.882337 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:09:54.907819 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.907796 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:09:54.968989 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:54.968913 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:55.032864 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.032835 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:09:55.034136 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.034122 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:09:55.095107 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.095077 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:09:55.305585 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:55.305416 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4357665f1982cbd14d16f18df231ee45.slice/crio-b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114 WatchSource:0}: Error finding container b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114: Status 404 returned error can't find the container with id b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114 Nov 03 16:09:55.312495 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.312477 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:09:55.418130 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.418097 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:09:55.540111 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.540082 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:09:55.544382 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.544313 2572 apiserver.go:52] "Watching apiserver" Nov 03 16:09:55.568962 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.568932 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 16:09:55.598828 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.598809 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:09:55.662511 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.662488 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:09:55.662826 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.662807 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-kv69g","openshift-network-operator/iptables-alerter-x86gk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd","openshift-ovn-kubernetes/ovnkube-node-gsdp4","kube-system/global-pull-secret-syncer-5dbjc","kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal","openshift-multus/multus-8r945","openshift-multus/network-metrics-daemon-fddzx","openshift-network-diagnostics/network-check-target-67tfw","kube-system/konnectivity-agent-hm72k","openshift-cluster-node-tuning-operator/tuned-54n6m","openshift-dns/node-resolver-xt54q","openshift-image-registry/node-ca-bcgxn"] Nov 03 16:09:55.666643 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.666627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:55.666708 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.666694 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:09:55.668797 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.668783 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.671152 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.671134 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.674057 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.674004 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.676629 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.676606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.676798 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.676664 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:09:55.678939 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.678919 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.681398 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.681377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:55.681482 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.681439 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:09:55.681863 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.681841 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-registration-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.681940 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.681870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7648n\" (UniqueName: \"kubernetes.io/projected/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kube-api-access-7648n\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.681940 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.681888 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecdd9d33-7747-417b-bff0-688639640a48-hosts-file\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.681940 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.681905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6q9\" (UniqueName: \"kubernetes.io/projected/ecdd9d33-7747-417b-bff0-688639640a48-kube-api-access-kr6q9\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.682111 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.681968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-socket-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.682111 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.681992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-device-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.682111 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.682014 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.682111 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.682044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-sys-fs\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.682111 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.682060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:55.682111 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.682104 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecdd9d33-7747-417b-bff0-688639640a48-tmp-dir\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.682463 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.682150 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.683979 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.683962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8r945" Nov 03 16:09:55.686479 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.686461 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.688916 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.688894 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.691382 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.691361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:55.693945 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.693925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.712388 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.712343 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"fed4b634a3c30570b86550ce73c63486","Type":"ContainerStarted","Data":"87c40c67d97f6d4f229fe3b64865f4d0bb39bf93f4f9519a8eea8f9fd8e302d8"} Nov 03 16:09:55.713326 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.713299 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"4357665f1982cbd14d16f18df231ee45","Type":"ContainerStarted","Data":"b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114"} Nov 03 16:09:55.730025 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.729998 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:09:55.730119 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.730038 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:09:55.730175 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.730137 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4xv66\"" Nov 03 16:09:55.730409 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.730392 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:09:55.730739 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.730711 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:09:55.730993 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.730974 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:09:55.731224 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.731209 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x72k5\"" Nov 03 16:09:55.733428 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.733410 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:09:55.733522 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.733446 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:09:55.733947 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.733931 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:09:55.734025 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.733982 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:09:55.734076 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.734024 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:09:55.734076 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.734033 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxk2h\"" Nov 03 16:09:55.734076 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.734032 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:09:55.738621 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.738605 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:09:55.738685 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.738639 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:09:55.738685 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.738645 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:09:55.738824 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.738810 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kkrgk\"" Nov 03 16:09:55.743795 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.743780 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:09:55.743795 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.743787 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:09:55.743901 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.743795 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:09:55.743942 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.743935 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:09:55.744039 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.744027 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xzbs4\"" Nov 03 16:09:55.745675 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.745660 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:09:55.770470 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.770454 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:09:55.779545 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.779525 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:09:55.782965 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.782947 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-node-log\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783044 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.782981 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.783044 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrdq\" (UniqueName: \"kubernetes.io/projected/c0c67fab-535b-4a8e-a336-ca475185c3dd-kube-api-access-ltrdq\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.783044 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783035 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-cni-binary-copy\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783151 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783094 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-netns\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783185 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-multus-certs\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783251 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecdd9d33-7747-417b-bff0-688639640a48-hosts-file\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.783304 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6q9\" (UniqueName: \"kubernetes.io/projected/ecdd9d33-7747-417b-bff0-688639640a48-kube-api-access-kr6q9\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.783304 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.783382 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783308 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783382 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783312 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecdd9d33-7747-417b-bff0-688639640a48-hosts-file\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.783382 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-cnibin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783382 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783365 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-ovn\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783518 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6md\" (UniqueName: \"kubernetes.io/projected/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-kube-api-access-jn6md\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.783518 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0c67fab-535b-4a8e-a336-ca475185c3dd-host\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.783518 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-conf-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783518 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783505 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-sys-fs\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.783673 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-slash\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783673 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-conf\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.783673 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783575 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-sys-fs\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.783673 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-tuned\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.783673 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-tmp\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.783673 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783639 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-system-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783673 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783660 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-systemd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783676 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-etc-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-netd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-modprobe-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-kubernetes\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783793 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-multus-daemon-config\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/289d2a22-071a-436a-afa6-8bfecabaa60c-host-slash\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783858 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-log-socket\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783913 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwd77\" (UniqueName: \"kubernetes.io/projected/43230f8a-cf14-4872-b51a-a4f675f947fb-kube-api-access-qwd77\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:55.783978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783962 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0c67fab-535b-4a8e-a336-ca475185c3dd-serviceca\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.783985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56sh\" (UniqueName: \"kubernetes.io/projected/289d2a22-071a-436a-afa6-8bfecabaa60c-kube-api-access-c56sh\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecdd9d33-7747-417b-bff0-688639640a48-tmp-dir\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-registration-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784158 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-registration-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784180 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-socket-dir-parent\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784228 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-k8s-cni-cncf-io\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784251 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skprg\" (UniqueName: \"kubernetes.io/projected/61daa065-4042-47db-add5-514189226920-kube-api-access-skprg\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784283 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/289d2a22-071a-436a-afa6-8bfecabaa60c-iptables-alerter-script\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-device-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784361 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784399 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-device-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-netns\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysconfig\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784460 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-systemd\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784474 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecdd9d33-7747-417b-bff0-688639640a48-tmp-dir\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784481 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784492 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9gv4t\"" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784491 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tk7qx\"" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784517 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784526 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-kubelet\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784587 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjnh\" (UniqueName: \"kubernetes.io/projected/8540df11-165c-4b01-b8a3-4042a8f6fdab-kube-api-access-6fjnh\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9189a7df-aae8-4728-be2b-569f2cffebc3-agent-certs\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784628 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7648n\" (UniqueName: \"kubernetes.io/projected/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kube-api-access-7648n\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784768 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784797 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xhvl5\"" Nov 03 16:09:55.784866 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784809 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4vrm9\"" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9189a7df-aae8-4728-be2b-569f2cffebc3-konnectivity-ca\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-sys\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-socket-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784926 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43230f8a-cf14-4872-b51a-a4f675f947fb-ovn-node-metrics-cert\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784947 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp64h\" (UniqueName: \"kubernetes.io/projected/507f7522-ec6f-4e48-b362-68e1c9dafeb8-kube-api-access-jp64h\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.784964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-dbus\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785005 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-socket-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785000 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-host\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-config\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-env-overrides\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785146 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-run\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785185 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-systemd-units\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.785594 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785262 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-bin\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785291 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-script-lib\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-system-cni-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785361 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-lib-modules\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785409 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-kubelet-config\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-hostroot\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785457 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-var-lib-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cnibin\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-bin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785527 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-kubelet\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-etc-kubernetes\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785577 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-var-lib-kubelet\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.786082 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.785625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-multus\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.786559 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786544 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:09:55.786623 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786576 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:09:55.786623 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786582 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:09:55.786701 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786616 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:09:55.786701 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786622 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:09:55.786701 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786645 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:09:55.786701 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786626 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:09:55.786701 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.786674 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kvdpg" Nov 03 16:09:55.868699 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.868614 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:09:55.868699 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.868644 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:09:55.868699 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.868655 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:55.868905 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.868732 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:56.368701443 +0000 UTC m=+3.196811814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:55.870192 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.870172 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:09:55.873899 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.873875 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6q9\" (UniqueName: \"kubernetes.io/projected/ecdd9d33-7747-417b-bff0-688639640a48-kube-api-access-kr6q9\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.874005 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.873878 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7648n\" (UniqueName: \"kubernetes.io/projected/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kube-api-access-7648n\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.885981 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.885953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-kubelet\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886035 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.885987 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjnh\" (UniqueName: \"kubernetes.io/projected/8540df11-165c-4b01-b8a3-4042a8f6fdab-kube-api-access-6fjnh\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:55.886035 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9189a7df-aae8-4728-be2b-569f2cffebc3-agent-certs\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:55.886101 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886039 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.886101 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886064 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-kubelet\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886116 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9189a7df-aae8-4728-be2b-569f2cffebc3-konnectivity-ca\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:55.886161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-sys\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.886319 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886319 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886206 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.886319 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43230f8a-cf14-4872-b51a-a4f675f947fb-ovn-node-metrics-cert\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886319 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp64h\" (UniqueName: \"kubernetes.io/projected/507f7522-ec6f-4e48-b362-68e1c9dafeb8-kube-api-access-jp64h\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.886319 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-dbus\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.886319 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-host\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-config\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-env-overrides\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886369 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-run\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886381 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886401 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-systemd-units\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-bin\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-script-lib\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886456 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-dbus\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-system-cni-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-lib-modules\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-kubelet-config\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886533 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-hostroot\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886534 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-systemd-units\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886540 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-host\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-var-lib-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.886599 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cnibin\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-bin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-kubelet\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886707 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-etc-kubernetes\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886741 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886768 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-var-lib-kubelet\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886801 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-multus\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886812 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9189a7df-aae8-4728-be2b-569f2cffebc3-konnectivity-ca\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-node-log\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886905 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-var-lib-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrdq\" (UniqueName: \"kubernetes.io/projected/c0c67fab-535b-4a8e-a336-ca475185c3dd-kube-api-access-ltrdq\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886927 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-run\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886935 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-env-overrides\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.886991 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cnibin\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887001 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-system-cni-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887038 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-bin\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.887451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887064 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-multus\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887076 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-hostroot\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887108 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-cni-binary-copy\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887126 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-lib-modules\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887135 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-netns\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.887129 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887150 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887168 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-etc-kubernetes\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887168 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-node-log\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887163 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-bin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-multus-certs\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.887275 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:09:56.387235849 +0000 UTC m=+3.215346224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887271 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-multus-certs\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887274 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-netns\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887325 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-sys\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887371 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-kubelet-config\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-config\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.888452 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887541 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-kubelet\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887571 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-var-lib-kubelet\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-cnibin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-ovn\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6md\" (UniqueName: \"kubernetes.io/projected/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-kube-api-access-jn6md\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887799 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0c67fab-535b-4a8e-a336-ca475185c3dd-host\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887844 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887852 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-conf-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887857 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-cnibin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887850 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-ovn\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887799 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-script-lib\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887891 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-conf-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887946 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-cni-binary-copy\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887953 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0c67fab-535b-4a8e-a336-ca475185c3dd-host\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.887989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-slash\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-conf\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889249 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-tuned\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888061 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-slash\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-tmp\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-system-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-systemd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888142 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-conf\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-etc-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-netd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888250 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-modprobe-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-kubernetes\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888270 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-system-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888272 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888181 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-systemd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-multus-daemon-config\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888358 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/289d2a22-071a-436a-afa6-8bfecabaa60c-host-slash\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888369 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-netd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888387 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-modprobe-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-log-socket\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.889966 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888422 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-log-socket\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888422 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd77\" (UniqueName: \"kubernetes.io/projected/43230f8a-cf14-4872-b51a-a4f675f947fb-kube-api-access-qwd77\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888481 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/289d2a22-071a-436a-afa6-8bfecabaa60c-host-slash\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888509 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-kubernetes\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888455 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.888555 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0c67fab-535b-4a8e-a336-ca475185c3dd-serviceca\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:55.888610 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:09:56.388592177 +0000 UTC m=+3.216702551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888589 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56sh\" (UniqueName: \"kubernetes.io/projected/289d2a22-071a-436a-afa6-8bfecabaa60c-kube-api-access-c56sh\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888647 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-etc-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-socket-dir-parent\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-k8s-cni-cncf-io\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888749 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skprg\" (UniqueName: \"kubernetes.io/projected/61daa065-4042-47db-add5-514189226920-kube-api-access-skprg\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/289d2a22-071a-436a-afa6-8bfecabaa60c-iptables-alerter-script\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.890504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888807 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888814 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-netns\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888845 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-multus-daemon-config\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888861 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-netns\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysconfig\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888906 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysconfig\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-systemd\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888958 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-systemd\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888964 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-k8s-cni-cncf-io\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.888775 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-socket-dir-parent\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.889773 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43230f8a-cf14-4872-b51a-a4f675f947fb-ovn-node-metrics-cert\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.889863 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0c67fab-535b-4a8e-a336-ca475185c3dd-serviceca\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.889925 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9189a7df-aae8-4728-be2b-569f2cffebc3-agent-certs\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.889994 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/289d2a22-071a-436a-afa6-8bfecabaa60c-iptables-alerter-script\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.890769 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-tmp\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.891029 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.890965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-tuned\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.954243 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.954203 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrdq\" (UniqueName: \"kubernetes.io/projected/c0c67fab-535b-4a8e-a336-ca475185c3dd-kube-api-access-ltrdq\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:55.955065 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.955039 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56sh\" (UniqueName: \"kubernetes.io/projected/289d2a22-071a-436a-afa6-8bfecabaa60c-kube-api-access-c56sh\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:55.955065 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.955062 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skprg\" (UniqueName: \"kubernetes.io/projected/61daa065-4042-47db-add5-514189226920-kube-api-access-skprg\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:09:55.955212 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.955191 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjnh\" (UniqueName: \"kubernetes.io/projected/8540df11-165c-4b01-b8a3-4042a8f6fdab-kube-api-access-6fjnh\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:55.955250 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.955205 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp64h\" (UniqueName: \"kubernetes.io/projected/507f7522-ec6f-4e48-b362-68e1c9dafeb8-kube-api-access-jp64h\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:55.955511 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.955495 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6md\" (UniqueName: \"kubernetes.io/projected/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-kube-api-access-jn6md\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:55.955924 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.955908 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd77\" (UniqueName: \"kubernetes.io/projected/43230f8a-cf14-4872-b51a-a4f675f947fb-kube-api-access-qwd77\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:55.982731 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.982708 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xt54q" Nov 03 16:09:55.992432 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:55.992413 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:09:55.992680 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:55.992659 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdd9d33_7747_417b_bff0_688639640a48.slice/crio-28b1f276c9a8cc016ccc7119edebc5f02f396ad024aac0fdad27a3009cf07301 WatchSource:0}: Error finding container 28b1f276c9a8cc016ccc7119edebc5f02f396ad024aac0fdad27a3009cf07301: Status 404 returned error can't find the container with id 28b1f276c9a8cc016ccc7119edebc5f02f396ad024aac0fdad27a3009cf07301 Nov 03 16:09:56.001313 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.001288 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc9fe0f_08e8_4cb1_bab7_ace0f47412c3.slice/crio-fe7ec78289e3c0d5c945b9ea4d3999bf62c7611201c21f4d95150d6516e442a7 WatchSource:0}: Error finding container fe7ec78289e3c0d5c945b9ea4d3999bf62c7611201c21f4d95150d6516e442a7: Status 404 returned error can't find the container with id fe7ec78289e3c0d5c945b9ea4d3999bf62c7611201c21f4d95150d6516e442a7 Nov 03 16:09:56.007534 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.007518 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:09:56.008635 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.008622 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8r945" Nov 03 16:09:56.015699 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.015677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:09:56.017897 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.017877 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:09:56.018305 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.018281 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c67fab_535b_4a8e_a336_ca475185c3dd.slice/crio-a4d113c1a2471e4a44779b05c6a78480ce7c8147b02d073e989def2314140f2c WatchSource:0}: Error finding container a4d113c1a2471e4a44779b05c6a78480ce7c8147b02d073e989def2314140f2c: Status 404 returned error can't find the container with id a4d113c1a2471e4a44779b05c6a78480ce7c8147b02d073e989def2314140f2c Nov 03 16:09:56.020735 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.020712 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61daa065_4042_47db_add5_514189226920.slice/crio-446f06d2e92ad81b4ec59682f6b21398563e75b42d49440ea4b7342652856a0b WatchSource:0}: Error finding container 446f06d2e92ad81b4ec59682f6b21398563e75b42d49440ea4b7342652856a0b: Status 404 returned error can't find the container with id 446f06d2e92ad81b4ec59682f6b21398563e75b42d49440ea4b7342652856a0b Nov 03 16:09:56.026928 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.026905 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43230f8a_cf14_4872_b51a_a4f675f947fb.slice/crio-ae082eed5008916f48815573bdc06e2bb3521ac656ac4c5269ff2348815626c6 WatchSource:0}: Error finding container ae082eed5008916f48815573bdc06e2bb3521ac656ac4c5269ff2348815626c6: Status 404 returned error can't find the container with id ae082eed5008916f48815573bdc06e2bb3521ac656ac4c5269ff2348815626c6 Nov 03 16:09:56.028738 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.028718 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289d2a22_071a_436a_afa6_8bfecabaa60c.slice/crio-27547d86ebff3bbf4b258bacccf495303f620bfae8ba86730882b7bde3f8e921 WatchSource:0}: Error finding container 27547d86ebff3bbf4b258bacccf495303f620bfae8ba86730882b7bde3f8e921: Status 404 returned error can't find the container with id 27547d86ebff3bbf4b258bacccf495303f620bfae8ba86730882b7bde3f8e921 Nov 03 16:09:56.032126 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.032107 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:09:56.035364 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.035347 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:09:56.039301 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.039269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:09:56.041737 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.041705 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9189a7df_aae8_4728_be2b_569f2cffebc3.slice/crio-3940a1ec9ee29b1dbf2f584e8b94b030a938a43b492a0bf8996d34f977202cb6 WatchSource:0}: Error finding container 3940a1ec9ee29b1dbf2f584e8b94b030a938a43b492a0bf8996d34f977202cb6: Status 404 returned error can't find the container with id 3940a1ec9ee29b1dbf2f584e8b94b030a938a43b492a0bf8996d34f977202cb6 Nov 03 16:09:56.044270 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.044252 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0428d5_c0b3_45e7_8c92_9fdcdc798786.slice/crio-5705828aad1a625a542a39e274f45a856bfea792b4f16143d598810889615f67 WatchSource:0}: Error finding container 5705828aad1a625a542a39e274f45a856bfea792b4f16143d598810889615f67: Status 404 returned error can't find the container with id 5705828aad1a625a542a39e274f45a856bfea792b4f16143d598810889615f67 Nov 03 16:09:56.048234 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:09:56.048211 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507f7522_ec6f_4e48_b362_68e1c9dafeb8.slice/crio-a12181362c94a76b51bb5c5bcfe89314053884f6ab2141085f7ab9036a2a26be WatchSource:0}: Error finding container a12181362c94a76b51bb5c5bcfe89314053884f6ab2141085f7ab9036a2a26be: Status 404 returned error can't find the container with id a12181362c94a76b51bb5c5bcfe89314053884f6ab2141085f7ab9036a2a26be Nov 03 16:09:56.393131 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.393094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:56.393131 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.393137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.393163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393287 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393310 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393332 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393347 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393351 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:09:57.393336144 +0000 UTC m=+4.221446501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393355 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393390 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:09:57.393378847 +0000 UTC m=+4.221489205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:56.393418 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:56.393402 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:57.393395584 +0000 UTC m=+4.221505941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:56.716320 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.716236 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerStarted","Data":"a12181362c94a76b51bb5c5bcfe89314053884f6ab2141085f7ab9036a2a26be"} Nov 03 16:09:56.717226 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.717180 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54n6m" event={"ID":"4c0428d5-c0b3-45e7-8c92-9fdcdc798786","Type":"ContainerStarted","Data":"5705828aad1a625a542a39e274f45a856bfea792b4f16143d598810889615f67"} Nov 03 16:09:56.718256 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.718232 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hm72k" event={"ID":"9189a7df-aae8-4728-be2b-569f2cffebc3","Type":"ContainerStarted","Data":"3940a1ec9ee29b1dbf2f584e8b94b030a938a43b492a0bf8996d34f977202cb6"} Nov 03 16:09:56.719320 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.719303 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"ae082eed5008916f48815573bdc06e2bb3521ac656ac4c5269ff2348815626c6"} Nov 03 16:09:56.720266 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.720247 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bcgxn" event={"ID":"c0c67fab-535b-4a8e-a336-ca475185c3dd","Type":"ContainerStarted","Data":"a4d113c1a2471e4a44779b05c6a78480ce7c8147b02d073e989def2314140f2c"} Nov 03 16:09:56.721345 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.721329 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" event={"ID":"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3","Type":"ContainerStarted","Data":"fe7ec78289e3c0d5c945b9ea4d3999bf62c7611201c21f4d95150d6516e442a7"} Nov 03 16:09:56.722375 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.722359 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x86gk" event={"ID":"289d2a22-071a-436a-afa6-8bfecabaa60c","Type":"ContainerStarted","Data":"27547d86ebff3bbf4b258bacccf495303f620bfae8ba86730882b7bde3f8e921"} Nov 03 16:09:56.723374 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.723358 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8r945" event={"ID":"61daa065-4042-47db-add5-514189226920","Type":"ContainerStarted","Data":"446f06d2e92ad81b4ec59682f6b21398563e75b42d49440ea4b7342652856a0b"} Nov 03 16:09:56.724427 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.724408 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xt54q" event={"ID":"ecdd9d33-7747-417b-bff0-688639640a48","Type":"ContainerStarted","Data":"28b1f276c9a8cc016ccc7119edebc5f02f396ad024aac0fdad27a3009cf07301"} Nov 03 16:09:56.790307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.790259 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:04:55 +0000 UTC" deadline="2028-06-30 11:49:38.362930315 +0000 UTC" Nov 03 16:09:56.790307 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:56.790301 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14515h39m41.572632787s" Nov 03 16:09:57.401200 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.401155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:57.401400 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.401220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:57.401400 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.401288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:57.401505 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401426 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:57.401505 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401479 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:09:59.401465166 +0000 UTC m=+6.229575538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:57.401804 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401789 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:09:57.401873 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401808 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:09:57.401873 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401818 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:57.401873 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401851 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:59.401840518 +0000 UTC m=+6.229950889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:57.402078 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401906 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:57.402078 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.401930 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:09:59.401921587 +0000 UTC m=+6.230031958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:57.716541 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.716457 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:57.717098 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.716628 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:09:57.717098 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.717080 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:57.717276 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.717186 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:09:57.720486 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.720413 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:57.720641 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:57.720613 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:09:57.791304 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.791208 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:04:55 +0000 UTC" deadline="2028-06-19 17:51:20.026047334 +0000 UTC" Nov 03 16:09:57.791304 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:57.791299 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14257h41m22.234755196s" Nov 03 16:09:59.421618 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:59.421565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:59.422125 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:59.421631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:59.422125 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:59.421678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:59.422125 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.421792 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:09:59.422125 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.421805 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:09:59.422125 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.421814 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:59.422125 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.421857 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:03.42184349 +0000 UTC m=+10.249953861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:09:59.422493 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.422164 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:59.422493 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.422198 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:10:03.422188182 +0000 UTC m=+10.250298553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:09:59.422493 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.422239 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:59.422493 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.422273 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:10:03.422261204 +0000 UTC m=+10.250371576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:09:59.709387 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:59.709300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:09:59.709569 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.709443 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:09:59.709879 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:59.709854 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:09:59.709990 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.709968 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:09:59.710048 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:09:59.710036 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:09:59.710207 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:09:59.710125 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:01.714398 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.714149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:01.714679 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:01.714533 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:01.714679 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.714608 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:01.714750 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:01.714696 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:01.714750 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.714743 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:01.714855 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:01.714831 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:01.747906 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.747878 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" event={"ID":"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3","Type":"ContainerStarted","Data":"9acd72799dfd030df98fbaae72d64c42cf6d67ee8445cffd8233f68ea29a09c6"} Nov 03 16:10:01.752598 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.752575 2572 generic.go:356] "Generic (PLEG): container finished" podID="fed4b634a3c30570b86550ce73c63486" containerID="6a4d9a0565009bd7244eb01bca4f02fde4d715b8f49b05adc1b98ca11194c5ad" exitCode=0 Nov 03 16:10:01.752776 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.752748 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"fed4b634a3c30570b86550ce73c63486","Type":"ContainerDied","Data":"6a4d9a0565009bd7244eb01bca4f02fde4d715b8f49b05adc1b98ca11194c5ad"} Nov 03 16:10:01.759490 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.759462 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"4357665f1982cbd14d16f18df231ee45","Type":"ContainerStarted","Data":"00d7d75730dfe4f13e0cc9166291e3f3939cc2d7c1e17f86e0da0daea084e5d2"} Nov 03 16:10:01.763670 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.763642 2572 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="2160ead286f403d511ffe0a82f4587943e13a788a871fee07cfd34dc29a689b9" exitCode=0 Nov 03 16:10:01.763764 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:01.763684 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerDied","Data":"2160ead286f403d511ffe0a82f4587943e13a788a871fee07cfd34dc29a689b9"} Nov 03 16:10:02.122355 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:02.122258 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" podStartSLOduration=7.122240036 podStartE2EDuration="7.122240036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:01.996047255 +0000 UTC m=+8.824157631" watchObservedRunningTime="2026-11-03 16:10:02.122240036 +0000 UTC m=+8.950350434" Nov 03 16:10:02.770356 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:02.770196 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hm72k" event={"ID":"9189a7df-aae8-4728-be2b-569f2cffebc3","Type":"ContainerStarted","Data":"3e945caade1faceeaa3e570d5dbf835d3705146bcd68ae09c263416dcb6daa81"} Nov 03 16:10:02.773135 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:02.773099 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8r945" event={"ID":"61daa065-4042-47db-add5-514189226920","Type":"ContainerStarted","Data":"5126f39337fe32c995b9a7d0d6f39930b33c77351e08ea79a1b33e0388ea31ea"} Nov 03 16:10:02.778451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:02.778416 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"fed4b634a3c30570b86550ce73c63486","Type":"ContainerStarted","Data":"e6e0f76ea250eb1acc1c8b9a64c61e373c124bc926aed1cf77e78f2e47d7b2d4"} Nov 03 16:10:03.015565 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.015460 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" podStartSLOduration=8.015441483 podStartE2EDuration="8.015441483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:03.014995349 +0000 UTC m=+9.843105728" watchObservedRunningTime="2026-11-03 16:10:03.015441483 +0000 UTC m=+9.843551862" Nov 03 16:10:03.015845 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.015807 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hm72k" podStartSLOduration=3.408345176 podStartE2EDuration="9.015791764s" totalImagesPullingTime="5.607446588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:02.894443849 +0000 UTC m=+9.722554227" watchObservedRunningTime="2026-11-03 16:10:03.015791764 +0000 UTC m=+9.843902141" Nov 03 16:10:03.140221 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.140150 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8r945" podStartSLOduration=2.931390415 podStartE2EDuration="9.140137786s" totalImagesPullingTime="6.208747371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:03.139711806 +0000 UTC m=+9.967822183" watchObservedRunningTime="2026-11-03 16:10:03.140137786 +0000 UTC m=+9.968248163" Nov 03 16:10:03.457524 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.457429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:03.457524 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.457481 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:03.457524 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.457521 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457609 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457613 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457640 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457654 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457659 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:10:11.457641196 +0000 UTC m=+18.285751570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457703 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457706 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:11.45769041 +0000 UTC m=+18.285800771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:03.457729 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.457729 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:10:11.457718828 +0000 UTC m=+18.285829204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:03.709386 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.709313 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:03.709548 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.709416 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:03.709818 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.709803 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:03.709892 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.709878 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:03.709953 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.709943 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:03.710008 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:03.709995 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:03.781463 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.781411 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bcgxn" event={"ID":"c0c67fab-535b-4a8e-a336-ca475185c3dd","Type":"ContainerStarted","Data":"1a83edc669a9e8790666848a1504f9de21b54e8719c808097bd1d6284438b917"} Nov 03 16:10:03.783194 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.783102 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xt54q" event={"ID":"ecdd9d33-7747-417b-bff0-688639640a48","Type":"ContainerStarted","Data":"0fe5acbd78431e5400fe81432138a310ce1aa486edf3343ef2f474d87c911c53"} Nov 03 16:10:03.902794 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:03.902738 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bcgxn" podStartSLOduration=3.193913174 podStartE2EDuration="9.902723274s" totalImagesPullingTime="6.7088101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:03.902144173 +0000 UTC m=+10.730254550" watchObservedRunningTime="2026-11-03 16:10:03.902723274 +0000 UTC m=+10.730833696" Nov 03 16:10:04.023107 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:04.023008 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xt54q" podStartSLOduration=3.326014934 podStartE2EDuration="10.02298974s" totalImagesPullingTime="6.696974806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:04.022749593 +0000 UTC m=+10.850859970" watchObservedRunningTime="2026-11-03 16:10:04.02298974 +0000 UTC m=+10.851100116" Nov 03 16:10:04.220258 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:04.220225 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:10:04.220889 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:04.220866 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:10:04.679255 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:04.679200 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:10:04.787614 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:04.787587 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" event={"ID":"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3","Type":"ContainerStarted","Data":"64a8357d621986cab85ba48786c6b94de618bef7f6197a4ca8c830a031eb8ff2"} Nov 03 16:10:04.787913 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:04.787890 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:10:04.788372 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:04.788349 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:10:05.654010 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.653914 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:10:04.679251158Z","UUID":"197fdc2e-93d2-43d6-a0be-a7c91bdfb8ba","Handler":null,"Name":"","Endpoint":""} Nov 03 16:10:05.655246 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.655222 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:10:05.655246 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.655249 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:10:05.711394 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.711326 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:05.711526 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.711399 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:05.711526 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.711423 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:05.711526 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:05.711507 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:05.711620 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:05.711545 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:05.711620 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:05.711608 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:05.791453 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.791418 2572 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="7ec88c02f1303eb549c2608ba3bebcf0226a11bf455126268c72d969dbc7ca58" exitCode=0 Nov 03 16:10:05.791891 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:05.791507 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerDied","Data":"7ec88c02f1303eb549c2608ba3bebcf0226a11bf455126268c72d969dbc7ca58"} Nov 03 16:10:06.797315 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:06.797027 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" event={"ID":"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3","Type":"ContainerStarted","Data":"235e66cd43c528adfcc4c6ca705066cd7f437e39b83cf07c90d6b1ba62c98588"} Nov 03 16:10:06.799010 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:06.798987 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x86gk" event={"ID":"289d2a22-071a-436a-afa6-8bfecabaa60c","Type":"ContainerStarted","Data":"cadf05471d203da54037a5d737712cbbae6b63b7e1a2bff6660f077802089583"} Nov 03 16:10:06.920294 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:06.920223 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" podStartSLOduration=2.971244913 podStartE2EDuration="12.920188949s" totalImagesPullingTime="9.948944036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:06.919982068 +0000 UTC m=+13.748092444" watchObservedRunningTime="2026-11-03 16:10:06.920188949 +0000 UTC m=+13.748299325" Nov 03 16:10:07.708729 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:07.708648 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:07.708729 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:07.708674 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:07.708930 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:07.708774 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:07.708930 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:07.708792 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:07.708930 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:07.708847 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:07.708930 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:07.708922 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:07.803351 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:07.803314 2572 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="0f63bd236223623fa651c982cfed1bd807ad0d3085ecc010ea6d65dfe9bd490d" exitCode=0 Nov 03 16:10:07.803788 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:07.803400 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerDied","Data":"0f63bd236223623fa651c982cfed1bd807ad0d3085ecc010ea6d65dfe9bd490d"} Nov 03 16:10:08.084355 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:08.084264 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x86gk" podStartSLOduration=5.333618735 podStartE2EDuration="14.08424956s" totalImagesPullingTime="8.750630825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:07.040139764 +0000 UTC m=+13.868250140" watchObservedRunningTime="2026-11-03 16:10:08.08424956 +0000 UTC m=+14.912359970" Nov 03 16:10:09.708896 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:09.708871 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:09.709473 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:09.708903 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:09.709473 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:09.708871 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:09.709473 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:09.708984 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:09.709473 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:09.709043 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:09.709473 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:09.709108 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:10.811296 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:10.811193 2572 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="d11c96055334defe42aabdc8299a61d172b3635c7fe1fba13fd57eb1af6373c3" exitCode=0 Nov 03 16:10:10.811296 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:10.811241 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerDied","Data":"d11c96055334defe42aabdc8299a61d172b3635c7fe1fba13fd57eb1af6373c3"} Nov 03 16:10:11.520239 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:11.520191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:11.520462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:11.520252 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:11.520462 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:11.520294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:11.520462 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520399 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:10:11.520462 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520428 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:10:11.520462 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520442 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:11.520673 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520500 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:27.520480969 +0000 UTC m=+34.348591338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:11.520673 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520405 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:11.520673 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520406 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:11.520804 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520744 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:10:27.5206733 +0000 UTC m=+34.348783669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:11.520842 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.520818 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:10:27.520795657 +0000 UTC m=+34.348906026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:11.709155 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:11.709120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:11.709155 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:11.709146 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:11.709460 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:11.709244 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:11.709460 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.709260 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:11.709460 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.709373 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:11.709603 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:11.709473 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:13.710165 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:13.709934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:13.710540 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:13.710102 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:13.710540 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:13.710284 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:13.710540 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:13.710114 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:13.710540 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:13.710390 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:13.710540 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:13.710432 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:13.821120 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:13.821031 2572 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="4e250fcfcb46d38445903394a2890223e5ab0f42fc1d9c0c4f55ed7f61100085" exitCode=0 Nov 03 16:10:13.821120 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:13.821090 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerDied","Data":"4e250fcfcb46d38445903394a2890223e5ab0f42fc1d9c0c4f55ed7f61100085"} Nov 03 16:10:14.826881 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:14.826843 2572 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="5fbab535fff0a5bb694f778bc6b6b368f9a4b415b3930955ff263d4588aca5ad" exitCode=0 Nov 03 16:10:14.827371 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:14.826888 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerDied","Data":"5fbab535fff0a5bb694f778bc6b6b368f9a4b415b3930955ff263d4588aca5ad"} Nov 03 16:10:15.711510 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:15.711487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:15.711693 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:15.711602 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:15.711693 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:15.711608 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:15.711693 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:15.711624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:15.711844 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:15.711721 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:15.711844 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:15.711802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:15.832267 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:15.832233 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv69g" event={"ID":"507f7522-ec6f-4e48-b362-68e1c9dafeb8","Type":"ContainerStarted","Data":"d565b57a11e872af5555348e2acad283d5aafc5ffc98dc704666593076e87165"} Nov 03 16:10:15.833633 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:15.833608 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54n6m" event={"ID":"4c0428d5-c0b3-45e7-8c92-9fdcdc798786","Type":"ContainerStarted","Data":"ff42149bfb60ac2e4e585c108cb95788793b8e9de1eed26d7c4f7a0fd3944c87"} Nov 03 16:10:15.962625 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:15.962561 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kv69g" podStartSLOduration=6.772825717 podStartE2EDuration="21.962547993s" totalImagesPullingTime="15.189722276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:15.961452009 +0000 UTC m=+22.789562387" watchObservedRunningTime="2026-11-03 16:10:15.962547993 +0000 UTC m=+22.790658382" Nov 03 16:10:16.084027 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:16.083912 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-54n6m" podStartSLOduration=2.665419215 podStartE2EDuration="22.083898571s" totalImagesPullingTime="19.418479356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:16.083371719 +0000 UTC m=+22.911482093" watchObservedRunningTime="2026-11-03 16:10:16.083898571 +0000 UTC m=+22.912008948" Nov 03 16:10:17.711504 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:17.711469 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:17.711948 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:17.711510 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:17.711948 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:17.711623 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:17.711948 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:17.711654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:17.711948 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:17.711750 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:17.711948 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:17.711807 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:19.709056 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:19.709023 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:19.709525 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:19.709031 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:19.709525 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:19.709123 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:19.709525 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:19.709238 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:19.709525 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:19.709289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:19.709525 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:19.709377 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:21.711726 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:21.711691 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:21.711726 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:21.711705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:21.712268 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:21.711803 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:21.712268 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:21.712144 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:21.712383 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:21.712269 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:21.712383 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:21.712353 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:23.709615 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:23.709585 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:23.710067 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:23.709678 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:23.710067 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:23.709752 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:23.710067 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:23.709845 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:23.710067 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:23.709889 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:23.710067 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:23.709966 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:25.708836 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:25.708794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:25.708836 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:25.708853 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:25.709363 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:25.708901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:25.709363 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:25.708999 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:25.709363 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:25.709044 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:25.709363 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:25.709156 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:27.048624 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.048590 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:27.293073 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.292914 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:27.529032 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.528995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.529041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.529071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529145 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529158 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529180 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529190 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529225 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:10:59.529189527 +0000 UTC m=+66.357299890 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529157 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529243 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:59.529235687 +0000 UTC m=+66.357346045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:27.529292 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.529265 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:10:59.529250164 +0000 UTC m=+66.357360524 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:27.711071 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.710989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:27.711071 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.711012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:27.711071 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.710989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:27.711316 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.711091 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:27.711316 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.711153 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:27.711316 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:27.711248 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:27.858839 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.858808 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"8af0974c28852c6ae4398e6c6ef656f06182dd763eebe242606a8bcb4f7c03c7"} Nov 03 16:10:27.858990 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.858898 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"64af5b38d309137ff911404e98c6f46940212d06b6451991a4af15a65efe9c4c"} Nov 03 16:10:27.858990 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.858929 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"9d4a116a3f9a79ae57b45ed7e7b08e0e06a7d8e3f501f03275d4422075495ef6"} Nov 03 16:10:27.858990 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.858964 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"3b52ee3add5aac42d9dc96174b27877d9b7e74f29803c056cc0e7ec58c9af014"} Nov 03 16:10:27.859078 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.858991 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"dcd5a15133f02fcddf66a0741aa8e0e70e896e6a871af059e5290026bed746f7"} Nov 03 16:10:27.859078 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:27.859018 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"f172eba4e79477041499325160dde99d34fa97c801893f3460792e36d0ac56e1"} Nov 03 16:10:28.959510 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:28.959471 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:29.478602 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:29.478569 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:29.711419 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:29.711338 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:29.711419 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:29.711351 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:29.711619 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:29.711439 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:29.711619 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:29.711452 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:29.711619 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:29.711527 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:29.711619 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:29.711608 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:29.723355 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:29.723325 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:29.867035 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:29.866998 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"dabe87d5ebc5007f43a41c5a3b429bd5657a1f4e52ed140a6723afe70616e5f4"} Nov 03 16:10:29.918393 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:29.918359 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:30.120299 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:30.120261 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:30.311684 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:30.311653 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:31.386091 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.386058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:31.708139 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.708113 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:31.708339 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.708113 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:31.708339 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:31.708254 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:31.708460 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:31.708329 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:31.708460 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.708120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:31.708550 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:31.708475 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:31.875080 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.875037 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" event={"ID":"43230f8a-cf14-4872-b51a-a4f675f947fb","Type":"ContainerStarted","Data":"97e64c22b17b10047edfb0da35ad38a462a997f2ccfd1ab7510e48ea77f2b99c"} Nov 03 16:10:31.875276 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.875246 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:10:31.875453 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.875426 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:10:31.875629 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.875498 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:10:31.890925 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.890903 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:10:31.891074 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:31.890993 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:10:32.173529 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:32.173477 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" podStartSLOduration=7.1337738569999996 podStartE2EDuration="38.173465733s" totalImagesPullingTime="31.039691876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:32.173269327 +0000 UTC m=+39.001379705" watchObservedRunningTime="2026-11-03 16:10:32.173465733 +0000 UTC m=+39.001576110" Nov 03 16:10:32.204734 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:32.204698 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:33.210477 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:33.210450 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xt54q_ecdd9d33-7747-417b-bff0-688639640a48/dns-node-resolver/0.log" Nov 03 16:10:33.631343 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:33.631310 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:33.708906 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:33.708878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:33.709064 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:33.708965 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:33.709064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:33.708983 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:33.709064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:33.709011 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:33.709496 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:33.709452 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:33.709617 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:33.709587 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:33.753274 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:33.753246 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bcgxn_c0c67fab-535b-4a8e-a336-ca475185c3dd/node-ca/0.log" Nov 03 16:10:34.206014 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:34.205983 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:34.434537 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:34.434506 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:34.652135 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:34.652097 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:35.576338 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:35.576300 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:35.709074 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:35.709042 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:35.709239 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:35.709102 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:35.709239 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:35.709106 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:35.709239 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:35.709190 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:35.709344 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:35.709298 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:35.709378 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:35.709356 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:35.785058 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:35.785016 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:36.013492 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:36.013451 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:36.209315 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:36.209283 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:36.395794 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:36.395713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:36.582743 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:36.582713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:37.634031 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:37.634000 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:37.708384 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:37.708345 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:37.708564 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:37.708354 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:37.708564 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:37.708478 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:37.708679 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:37.708566 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:37.708679 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:37.708355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:37.708679 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:37.708671 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:37.858177 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:37.858139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:38.064738 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:38.064707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:38.254340 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:38.254310 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:38.452736 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:38.452662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:39.143159 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:39.143126 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:39.328943 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:39.328911 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:39.660556 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:39.660526 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:39.709174 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:39.709145 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:39.709324 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:39.709181 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:39.709324 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:39.709149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:39.709324 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:39.709268 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:39.709441 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:39.709349 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:39.709441 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:39.709429 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:39.846995 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:39.846965 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:40.041706 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:40.041669 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:40.232348 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:40.232160 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:40.451168 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:40.451067 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:41.630391 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:41.630356 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:41.708259 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:41.708230 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:41.708259 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:41.708253 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:41.708500 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:41.708261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:41.708500 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:41.708347 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:41.708500 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:41.708404 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:41.708500 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:41.708480 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:41.827226 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:41.827172 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:42.014768 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:42.014738 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:42.203102 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:42.203067 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:42.397723 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:42.397646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:42.599136 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:42.599110 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:42.848400 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:42.848371 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:43.368230 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:43.368180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:43.560355 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:43.560326 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:43.709699 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:43.709616 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:43.709699 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:43.709675 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:43.709905 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:43.709745 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:43.709905 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:43.709761 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:43.709905 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:43.709842 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:43.710062 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:43.709912 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:43.742088 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:43.742060 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:43.947530 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:43.947500 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:44.146932 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:44.146903 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:44.338493 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:44.338457 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:44.532671 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:44.532629 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:44.828624 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:44.828537 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:45.411890 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:45.411855 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:45.597137 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:45.597108 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:45.708528 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:45.708447 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:45.708698 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:45.708458 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:45.708698 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:45.708557 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:45.708698 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:45.708575 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:45.708698 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:45.708662 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:45.708835 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:45.708756 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:45.796604 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:45.796576 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:46.526988 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:46.526954 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:46.873538 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:46.873469 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:47.709032 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:47.708994 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:47.709032 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:47.709022 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:47.709499 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:47.708996 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:47.709499 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:47.709124 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:47.709499 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:47.709184 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:47.709499 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:47.709264 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:47.907880 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:47.907814 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:10:47.907880 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:47.907849 2572 generic.go:356] "Generic (PLEG): container finished" podID="61daa065-4042-47db-add5-514189226920" containerID="5126f39337fe32c995b9a7d0d6f39930b33c77351e08ea79a1b33e0388ea31ea" exitCode=1 Nov 03 16:10:47.908044 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:47.907879 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8r945" event={"ID":"61daa065-4042-47db-add5-514189226920","Type":"ContainerDied","Data":"5126f39337fe32c995b9a7d0d6f39930b33c77351e08ea79a1b33e0388ea31ea"} Nov 03 16:10:47.908238 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:47.908198 2572 scope.go:122] "RemoveContainer" containerID="5126f39337fe32c995b9a7d0d6f39930b33c77351e08ea79a1b33e0388ea31ea" Nov 03 16:10:48.911883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:48.911857 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:10:48.912377 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:48.911928 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8r945" event={"ID":"61daa065-4042-47db-add5-514189226920","Type":"ContainerStarted","Data":"b14eb0fe994b8c5db8a089f3df133bfab388618f4d1b9f90fcc8e07c01675a5f"} Nov 03 16:10:49.709172 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:49.709132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:49.709172 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:49.709161 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:49.709415 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:49.709278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:49.709415 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:49.709295 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:49.709415 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:49.709354 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:49.709530 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:49.709447 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:51.708766 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:51.708739 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:51.709324 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:51.708847 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:51.709324 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:51.708887 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:51.709324 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:51.708989 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:51.709324 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:51.709012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:51.709324 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:51.709095 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:53.709115 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:53.709086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:53.709485 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:53.709221 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:53.709485 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:53.709244 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:53.709485 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:53.709288 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:53.709485 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:53.709320 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:53.709485 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:53.709413 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:53.955302 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:53.955264 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:54.151800 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:54.151764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:54.534432 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:54.534399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:54.727822 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:54.727789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:55.427150 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:55.426979 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5dbjc"] Nov 03 16:10:55.427322 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:55.427272 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:55.427380 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:55.427361 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:55.430419 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:55.430384 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fddzx"] Nov 03 16:10:55.430572 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:55.430523 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:55.430675 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:55.430642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:55.431255 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:55.431233 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67tfw"] Nov 03 16:10:55.431353 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:55.431340 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:55.431459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:55.431434 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:56.612012 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:56.611984 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:56.809698 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:56.809665 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:57.708304 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:57.708276 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:57.708678 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:57.708326 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:57.708678 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:57.708339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:57.708678 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:57.708425 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:57.708678 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:57.708491 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:57.708678 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:57.708534 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:57.827479 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:57.827443 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:58.012720 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:58.012684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:58.222167 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:58.222138 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:58.443753 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:58.443684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:59.543989 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.543954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.544001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.544029 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544108 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544123 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544136 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544144 2572 projected.go:196] Error preparing data for projected volume kube-api-access-42gvz for pod openshift-network-diagnostics/network-check-target-67tfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544148 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544177 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret podName:ab478348-e9ee-4b58-a398-422dd7e03f0b nodeName:}" failed. No retries permitted until 2026-11-03 16:12:03.544160105 +0000 UTC m=+130.372270477 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret") pod "global-pull-secret-syncer-5dbjc" (UID: "ab478348-e9ee-4b58-a398-422dd7e03f0b") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544191 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz podName:caf14ab9-d061-4cfb-bc58-68ecf17a10a7 nodeName:}" failed. No retries permitted until 2026-11-03 16:12:03.54418554 +0000 UTC m=+130.372295894 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-42gvz" (UniqueName: "kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz") pod "network-check-target-67tfw" (UID: "caf14ab9-d061-4cfb-bc58-68ecf17a10a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:10:59.544459 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.544223 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs podName:8540df11-165c-4b01-b8a3-4042a8f6fdab nodeName:}" failed. No retries permitted until 2026-11-03 16:12:03.544197661 +0000 UTC m=+130.372308017 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs") pod "network-metrics-daemon-fddzx" (UID: "8540df11-165c-4b01-b8a3-4042a8f6fdab") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:10:59.708679 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.708596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:10:59.708679 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.708633 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:10:59.708879 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.708596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:10:59.708879 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.708726 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fddzx" podUID="8540df11-165c-4b01-b8a3-4042a8f6fdab" Nov 03 16:10:59.708879 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.708813 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67tfw" podUID="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" Nov 03 16:10:59.708980 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:10:59.708884 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5dbjc" podUID="ab478348-e9ee-4b58-a398-422dd7e03f0b" Nov 03 16:10:59.768943 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.768912 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:59.964172 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:10:59.964091 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:00.970995 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:00.970916 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeReady" Nov 03 16:11:00.971370 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:00.971050 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 16:11:01.091536 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.091497 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xg99b"] Nov 03 16:11:01.093493 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.093478 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.143630 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.143601 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wc6dg"] Nov 03 16:11:01.145457 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.145441 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qbst4"] Nov 03 16:11:01.145586 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.145570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.147374 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.147356 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.150105 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.150083 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zz2z\"" Nov 03 16:11:01.150231 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.150173 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:11:01.151096 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.151082 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:11:01.186711 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.186684 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xg99b"] Nov 03 16:11:01.204791 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.204766 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wc6dg"] Nov 03 16:11:01.208396 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.208376 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbst4"] Nov 03 16:11:01.250861 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.250825 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:11:01.251007 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.250896 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:11:01.251007 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.250937 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gswn6\"" Nov 03 16:11:01.251093 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.251006 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4pfhl\"" Nov 03 16:11:01.251848 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.251834 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:11:01.251848 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.251838 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:11:01.251952 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.251838 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:11:01.251952 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.251872 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:11:01.251952 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.251894 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:11:01.258097 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2218179a-fa77-4676-ad59-885f522a5039-config-volume\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.258172 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258112 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7da5e2a9-8f58-4ee1-8a00-c443413a249c-data-volume\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.258172 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxqv\" (UniqueName: \"kubernetes.io/projected/05b3881e-7078-461d-9c94-69d049ccc417-kube-api-access-8pxqv\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.258281 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czjl\" (UniqueName: \"kubernetes.io/projected/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-api-access-8czjl\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.258381 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258365 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b3881e-7078-461d-9c94-69d049ccc417-cert\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.258437 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2218179a-fa77-4676-ad59-885f522a5039-tmp-dir\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.258437 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.258534 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62d8\" (UniqueName: \"kubernetes.io/projected/2218179a-fa77-4676-ad59-885f522a5039-kube-api-access-j62d8\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.258588 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7da5e2a9-8f58-4ee1-8a00-c443413a249c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.258588 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258576 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2218179a-fa77-4676-ad59-885f522a5039-metrics-tls\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.258677 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.258608 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7da5e2a9-8f58-4ee1-8a00-c443413a249c-crio-socket\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.359628 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62d8\" (UniqueName: \"kubernetes.io/projected/2218179a-fa77-4676-ad59-885f522a5039-kube-api-access-j62d8\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.359813 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7da5e2a9-8f58-4ee1-8a00-c443413a249c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.359813 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359663 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2218179a-fa77-4676-ad59-885f522a5039-metrics-tls\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.359813 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7da5e2a9-8f58-4ee1-8a00-c443413a249c-crio-socket\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.359813 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2218179a-fa77-4676-ad59-885f522a5039-config-volume\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.359813 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7da5e2a9-8f58-4ee1-8a00-c443413a249c-data-volume\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.359813 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxqv\" (UniqueName: \"kubernetes.io/projected/05b3881e-7078-461d-9c94-69d049ccc417-kube-api-access-8pxqv\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.360097 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czjl\" (UniqueName: \"kubernetes.io/projected/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-api-access-8czjl\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.360097 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359837 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7da5e2a9-8f58-4ee1-8a00-c443413a249c-crio-socket\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.360097 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b3881e-7078-461d-9c94-69d049ccc417-cert\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.360097 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359913 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2218179a-fa77-4676-ad59-885f522a5039-tmp-dir\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.360097 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.359936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.360484 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.360463 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2218179a-fa77-4676-ad59-885f522a5039-config-volume\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.360562 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.360509 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2218179a-fa77-4676-ad59-885f522a5039-tmp-dir\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.360562 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.360552 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.360649 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.360623 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7da5e2a9-8f58-4ee1-8a00-c443413a249c-data-volume\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.363777 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.363753 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2218179a-fa77-4676-ad59-885f522a5039-metrics-tls\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.363871 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.363808 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7da5e2a9-8f58-4ee1-8a00-c443413a249c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.363912 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.363898 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b3881e-7078-461d-9c94-69d049ccc417-cert\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.421567 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.421532 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62d8\" (UniqueName: \"kubernetes.io/projected/2218179a-fa77-4676-ad59-885f522a5039-kube-api-access-j62d8\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.421567 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.421561 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxqv\" (UniqueName: \"kubernetes.io/projected/05b3881e-7078-461d-9c94-69d049ccc417-kube-api-access-8pxqv\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.421722 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.421533 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czjl\" (UniqueName: \"kubernetes.io/projected/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-api-access-8czjl\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.458883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.458851 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:11:01.466014 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.465987 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:11:01.704715 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.704541 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:01.708670 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.708652 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:11:01.708788 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.708652 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:11:01.708860 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.708652 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:11:01.765969 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.765944 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:11:01.766109 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.766092 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:11:01.766181 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.766163 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zsc2l\"" Nov 03 16:11:01.766285 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.766276 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:11:01.766376 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.766362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:11:01.766437 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.766414 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rqr4v\"" Nov 03 16:11:01.766490 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.766453 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7zlt\"" Nov 03 16:11:01.897273 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.897247 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wc6dg"] Nov 03 16:11:01.899134 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:01.899107 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b3881e_7078_461d_9c94_69d049ccc417.slice/crio-71b2223e985e4191587f83be652b00056d583876f94db1ba5ebc2b750823d2de WatchSource:0}: Error finding container 71b2223e985e4191587f83be652b00056d583876f94db1ba5ebc2b750823d2de: Status 404 returned error can't find the container with id 71b2223e985e4191587f83be652b00056d583876f94db1ba5ebc2b750823d2de Nov 03 16:11:01.899788 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.899769 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbst4"] Nov 03 16:11:01.901432 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:01.901408 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da5e2a9_8f58_4ee1_8a00_c443413a249c.slice/crio-9a7cc691cd382057664b54b18d4c13cf31888e63190e40eded9609f9e29f97c7 WatchSource:0}: Error finding container 9a7cc691cd382057664b54b18d4c13cf31888e63190e40eded9609f9e29f97c7: Status 404 returned error can't find the container with id 9a7cc691cd382057664b54b18d4c13cf31888e63190e40eded9609f9e29f97c7 Nov 03 16:11:01.938905 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.938875 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbst4" event={"ID":"7da5e2a9-8f58-4ee1-8a00-c443413a249c","Type":"ContainerStarted","Data":"9a7cc691cd382057664b54b18d4c13cf31888e63190e40eded9609f9e29f97c7"} Nov 03 16:11:01.939990 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:01.939969 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wc6dg" event={"ID":"05b3881e-7078-461d-9c94-69d049ccc417","Type":"ContainerStarted","Data":"71b2223e985e4191587f83be652b00056d583876f94db1ba5ebc2b750823d2de"} Nov 03 16:11:02.084413 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:02.084389 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xg99b"] Nov 03 16:11:02.086724 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:02.086697 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2218179a_fa77_4676_ad59_885f522a5039.slice/crio-d28b27f98f82bd7dd24b18d0f8f41eec8a911886466d382660e839f75c09ba04 WatchSource:0}: Error finding container d28b27f98f82bd7dd24b18d0f8f41eec8a911886466d382660e839f75c09ba04: Status 404 returned error can't find the container with id d28b27f98f82bd7dd24b18d0f8f41eec8a911886466d382660e839f75c09ba04 Nov 03 16:11:02.944241 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:02.944189 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbst4" event={"ID":"7da5e2a9-8f58-4ee1-8a00-c443413a249c","Type":"ContainerStarted","Data":"9020ab4a7dae19d7afe58612def466f3c70b91268774da640f42fbf535c8c531"} Nov 03 16:11:02.945155 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:02.945127 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xg99b" event={"ID":"2218179a-fa77-4676-ad59-885f522a5039","Type":"ContainerStarted","Data":"d28b27f98f82bd7dd24b18d0f8f41eec8a911886466d382660e839f75c09ba04"} Nov 03 16:11:03.894782 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:03.894576 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:11:03.950727 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:03.950684 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbst4" event={"ID":"7da5e2a9-8f58-4ee1-8a00-c443413a249c","Type":"ContainerStarted","Data":"4e2a17cd6fecafa7781cc14053dd6b0c20e0c6cd432a41b5285331b9b6163cd6"} Nov 03 16:11:03.965148 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:03.965111 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:04.249110 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:04.249070 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:04.438500 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:04.438468 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:05.047661 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.047633 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:05.592720 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.592689 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:05.816724 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.816651 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:05.958281 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.958242 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbst4" event={"ID":"7da5e2a9-8f58-4ee1-8a00-c443413a249c","Type":"ContainerStarted","Data":"53bc52e05fa304a2faf6cf9d18d3c79addfdf2116e5d00ddae24116bf4ead63f"} Nov 03 16:11:05.959561 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.959535 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wc6dg" event={"ID":"05b3881e-7078-461d-9c94-69d049ccc417","Type":"ContainerStarted","Data":"a94c7e7673c34d8c4108454c9e1cb6ce3c206fd081caec66e2b52cbf1277feb7"} Nov 03 16:11:05.961174 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.961152 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xg99b" event={"ID":"2218179a-fa77-4676-ad59-885f522a5039","Type":"ContainerStarted","Data":"668da287ca426be81d8049bce8125be74715af6d608865cb138da81a64b3b06a"} Nov 03 16:11:05.961309 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.961188 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xg99b" event={"ID":"2218179a-fa77-4676-ad59-885f522a5039","Type":"ContainerStarted","Data":"ca1952d70f45ac4e29da5304ed8085cf7ad99abb4ef1e17b8fb0eee1b5e780a4"} Nov 03 16:11:05.961309 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:05.961289 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:06.015782 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.015751 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:06.216365 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.216256 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:06.217278 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.217242 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qbst4" podStartSLOduration=1.686935978 podStartE2EDuration="5.217228446s" totalImagesPullingTime="3.530292468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:11:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:06.08855793 +0000 UTC m=+72.916668307" watchObservedRunningTime="2026-11-03 16:11:06.217228446 +0000 UTC m=+73.045338814" Nov 03 16:11:06.342628 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.342578 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xg99b" podStartSLOduration=2.47691061 podStartE2EDuration="5.342562404s" totalImagesPullingTime="2.865651794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:06.21975927 +0000 UTC m=+73.047869658" watchObservedRunningTime="2026-11-03 16:11:06.342562404 +0000 UTC m=+73.170672780" Nov 03 16:11:06.342814 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.342793 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wc6dg" podStartSLOduration=1.942536456 podStartE2EDuration="5.342786466s" totalImagesPullingTime="3.40025001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:06.342290056 +0000 UTC m=+73.170400436" watchObservedRunningTime="2026-11-03 16:11:06.342786466 +0000 UTC m=+73.170896843" Nov 03 16:11:06.418410 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.418382 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:06.759150 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.759113 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:06.938267 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.938238 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw"] Nov 03 16:11:06.940149 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.940134 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:06.964302 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.964269 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:06.990491 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.990463 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql"] Nov 03 16:11:06.992535 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:06.992520 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.005415 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.005500 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.005583 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvlq9\" (UniqueName: \"kubernetes.io/projected/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-api-access-mvlq9\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.005617 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005601 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.005784 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005762 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61a83e02-ce4d-4df6-bc29-368ec906215f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.005827 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3833015d-9c9e-49d7-8630-a4b118d57ab6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.005861 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005834 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.005861 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.005938 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.005919 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.006065 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.006049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5fdm\" (UniqueName: \"kubernetes.io/projected/3833015d-9c9e-49d7-8630-a4b118d57ab6-kube-api-access-w5fdm\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.015192 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.015145 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:11:07.015192 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.015180 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:11:07.015316 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.015226 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:11:07.042167 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.042135 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x292s\"" Nov 03 16:11:07.048125 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.048099 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:11:07.048272 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.048195 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 16:11:07.049507 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.049481 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw"] Nov 03 16:11:07.051102 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.051077 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql"] Nov 03 16:11:07.068619 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.068594 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hfllm"] Nov 03 16:11:07.070672 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.070657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.100493 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.100469 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:11:07.100658 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.100535 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h7tfb\"" Nov 03 16:11:07.100658 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.100571 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 16:11:07.100735 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.100695 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 16:11:07.107064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107119 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-textfile\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107119 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.107251 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvlq9\" (UniqueName: \"kubernetes.io/projected/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-api-access-mvlq9\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.107326 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107309 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhtmh\" (UniqueName: \"kubernetes.io/projected/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-kube-api-access-lhtmh\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107380 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3833015d-9c9e-49d7-8630-a4b118d57ab6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.107380 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107377 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-root\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107531 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107513 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.107589 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107543 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.107589 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107570 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107693 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107599 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fdm\" (UniqueName: \"kubernetes.io/projected/3833015d-9c9e-49d7-8630-a4b118d57ab6-kube-api-access-w5fdm\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.107693 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107623 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-sys\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107693 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-metrics-client-ca\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107693 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107675 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.107883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107705 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.107883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.107883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107759 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-wtmp\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-tls\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.107883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61a83e02-ce4d-4df6-bc29-368ec906215f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.107883 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.107873 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.108148 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.108065 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3833015d-9c9e-49d7-8630-a4b118d57ab6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.108225 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.108187 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61a83e02-ce4d-4df6-bc29-368ec906215f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.108373 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.108350 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.111587 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.111566 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.111683 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.111565 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.111683 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.111601 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.111804 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.111789 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.170056 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.170012 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvlq9\" (UniqueName: \"kubernetes.io/projected/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-api-access-mvlq9\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.170195 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.170023 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fdm\" (UniqueName: \"kubernetes.io/projected/3833015d-9c9e-49d7-8630-a4b118d57ab6-kube-api-access-w5fdm\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.175223 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.175181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:07.197882 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.197816 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:11:07.198256 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.198229 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:11:07.198361 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.198275 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:11:07.198361 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.198308 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m7glm\"" Nov 03 16:11:07.208339 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208313 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtmh\" (UniqueName: \"kubernetes.io/projected/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-kube-api-access-lhtmh\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208441 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-root\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208491 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208440 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-root\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208562 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208546 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208603 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-sys\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208603 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-metrics-client-ca\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208715 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-wtmp\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208715 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208632 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-tls\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208715 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208658 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-sys\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208715 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208674 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208715 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-textfile\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.208942 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208771 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-wtmp\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.209012 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.208997 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-textfile\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.209144 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.209126 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.209240 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.209190 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-metrics-client-ca\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.211054 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.211031 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.211180 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.211161 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-tls\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.252605 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.252573 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:11:07.272409 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.272339 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtmh\" (UniqueName: \"kubernetes.io/projected/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-kube-api-access-lhtmh\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.304666 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.304636 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:11:07.382533 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.382502 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:11:07.392247 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:07.392217 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234cc11d_353c_4f0c_ab3b_4616ddd0b26d.slice/crio-7a5bf72fdce77c3fe97c071e3a35ab348b54ec0f28dc84348b84a9b752a049ce WatchSource:0}: Error finding container 7a5bf72fdce77c3fe97c071e3a35ab348b54ec0f28dc84348b84a9b752a049ce: Status 404 returned error can't find the container with id 7a5bf72fdce77c3fe97c071e3a35ab348b54ec0f28dc84348b84a9b752a049ce Nov 03 16:11:07.539254 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.539151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:07.628482 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.628459 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw"] Nov 03 16:11:07.629865 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:07.629842 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3833015d_9c9e_49d7_8630_a4b118d57ab6.slice/crio-0710ffa9a941cc18f494e9dd2af469e43af2875eda5f24677845d7ec0c82d9b2 WatchSource:0}: Error finding container 0710ffa9a941cc18f494e9dd2af469e43af2875eda5f24677845d7ec0c82d9b2: Status 404 returned error can't find the container with id 0710ffa9a941cc18f494e9dd2af469e43af2875eda5f24677845d7ec0c82d9b2 Nov 03 16:11:07.692512 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.692490 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql"] Nov 03 16:11:07.697547 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:07.697493 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a83e02_ce4d_4df6_bc29_368ec906215f.slice/crio-ebf950b28b0574653f84097e73ed1ec5c049bed29fa96063cf4289dded008f6f WatchSource:0}: Error finding container ebf950b28b0574653f84097e73ed1ec5c049bed29fa96063cf4289dded008f6f: Status 404 returned error can't find the container with id ebf950b28b0574653f84097e73ed1ec5c049bed29fa96063cf4289dded008f6f Nov 03 16:11:07.751860 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.751824 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:07.967860 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.967779 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" event={"ID":"3833015d-9c9e-49d7-8630-a4b118d57ab6","Type":"ContainerStarted","Data":"1f2ae57c4618d5502104fee8f5a2907abb25fed575a556dcd74864044c903e3c"} Nov 03 16:11:07.967860 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.967831 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" event={"ID":"3833015d-9c9e-49d7-8630-a4b118d57ab6","Type":"ContainerStarted","Data":"5e10202af90fc96c5ce166cdaab1d7fbe67d9b0a88c060800018c8cadedb195b"} Nov 03 16:11:07.968034 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.967860 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" event={"ID":"3833015d-9c9e-49d7-8630-a4b118d57ab6","Type":"ContainerStarted","Data":"0710ffa9a941cc18f494e9dd2af469e43af2875eda5f24677845d7ec0c82d9b2"} Nov 03 16:11:07.968914 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.968891 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" event={"ID":"61a83e02-ce4d-4df6-bc29-368ec906215f","Type":"ContainerStarted","Data":"ebf950b28b0574653f84097e73ed1ec5c049bed29fa96063cf4289dded008f6f"} Nov 03 16:11:07.970025 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:07.970007 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfllm" event={"ID":"234cc11d-353c-4f0c-ab3b-4616ddd0b26d","Type":"ContainerStarted","Data":"7a5bf72fdce77c3fe97c071e3a35ab348b54ec0f28dc84348b84a9b752a049ce"} Nov 03 16:11:08.453857 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:08.453827 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:08.644832 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:08.644800 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:09.305428 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.305377 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:09.528287 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.528169 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:09.680132 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.680102 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b684bf768-ghn5f"] Nov 03 16:11:09.682670 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.682650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.717529 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.717498 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:09.729603 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729576 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e9344a7-9120-40ef-8671-5bc1fd4de206-metrics-client-ca\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.729718 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729616 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.729718 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.729835 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729754 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.729835 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729808 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-grpc-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.729938 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.729938 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729874 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.729938 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e9344a7-9120-40ef-8671-5bc1fd4de206-tmp\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.730083 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.729993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrll\" (UniqueName: \"kubernetes.io/projected/5e9344a7-9120-40ef-8671-5bc1fd4de206-kube-api-access-7xrll\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.739305 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.739269 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 16:11:09.739305 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.739277 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bqa94k5oj8mrd\"" Nov 03 16:11:09.739305 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.739279 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 16:11:09.789752 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.789721 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fkfj5\"" Nov 03 16:11:09.789896 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.789758 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 16:11:09.790090 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.790074 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 16:11:09.791166 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.791143 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b684bf768-ghn5f"] Nov 03 16:11:09.830997 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.830979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-grpc-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831080 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831080 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831060 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831250 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e9344a7-9120-40ef-8671-5bc1fd4de206-tmp\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831303 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrll\" (UniqueName: \"kubernetes.io/projected/5e9344a7-9120-40ef-8671-5bc1fd4de206-kube-api-access-7xrll\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831336 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e9344a7-9120-40ef-8671-5bc1fd4de206-metrics-client-ca\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831377 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831423 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831474 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831455 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831528 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e9344a7-9120-40ef-8671-5bc1fd4de206-tmp\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.831980 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.831956 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e9344a7-9120-40ef-8671-5bc1fd4de206-metrics-client-ca\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.833650 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.833623 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.833804 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.833787 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.833937 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.833919 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.833937 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.833930 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.834056 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.834041 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-grpc-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.839267 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.839238 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 16:11:09.843904 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.843884 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.892399 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.892374 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrll\" (UniqueName: \"kubernetes.io/projected/5e9344a7-9120-40ef-8671-5bc1fd4de206-kube-api-access-7xrll\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:09.984069 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.984040 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" event={"ID":"3833015d-9c9e-49d7-8630-a4b118d57ab6","Type":"ContainerStarted","Data":"dc2db7110e128d96a18e0283d14d3a1347d275860df938bd1d1e45bd7d1f031b"} Nov 03 16:11:09.986275 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.986250 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" event={"ID":"61a83e02-ce4d-4df6-bc29-368ec906215f","Type":"ContainerStarted","Data":"d4aca859dc1be63caf72ffd7a2a4de45aaf479a645018802230ba57c400cf318"} Nov 03 16:11:09.986358 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.986297 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" event={"ID":"61a83e02-ce4d-4df6-bc29-368ec906215f","Type":"ContainerStarted","Data":"a67b524e81f92fa9bc486f4cc06640961d60ddb5a9ece762fa21ac80cea0475a"} Nov 03 16:11:09.987564 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.987543 2572 generic.go:356] "Generic (PLEG): container finished" podID="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" containerID="5dacbcee14cc7f96b546deaa53217704e453e0f009dce2225b8a3db0dbb6cdd1" exitCode=0 Nov 03 16:11:09.987628 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:09.987587 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfllm" event={"ID":"234cc11d-353c-4f0c-ab3b-4616ddd0b26d","Type":"ContainerDied","Data":"5dacbcee14cc7f96b546deaa53217704e453e0f009dce2225b8a3db0dbb6cdd1"} Nov 03 16:11:10.001034 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:10.001014 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:10.114237 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:10.109436 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" podStartSLOduration=2.450640585 podStartE2EDuration="4.109416798s" totalImagesPullingTime="1.658776213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:10.108299377 +0000 UTC m=+76.936409756" watchObservedRunningTime="2026-11-03 16:11:10.109416798 +0000 UTC m=+76.937527180" Nov 03 16:11:10.436583 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:10.436551 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b684bf768-ghn5f"] Nov 03 16:11:10.437706 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:10.437682 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9344a7_9120_40ef_8671_5bc1fd4de206.slice/crio-f0cc589b8a988fb2e7c922b797125feaaa9fe29d96140442045106196f587069 WatchSource:0}: Error finding container f0cc589b8a988fb2e7c922b797125feaaa9fe29d96140442045106196f587069: Status 404 returned error can't find the container with id f0cc589b8a988fb2e7c922b797125feaaa9fe29d96140442045106196f587069 Nov 03 16:11:10.995109 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:10.995077 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" event={"ID":"61a83e02-ce4d-4df6-bc29-368ec906215f","Type":"ContainerStarted","Data":"428acfb251fe5ca70197f877616db0feab5d27bf49ee4e8db3d5bc28a2b940b0"} Nov 03 16:11:10.997351 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:10.997327 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfllm" event={"ID":"234cc11d-353c-4f0c-ab3b-4616ddd0b26d","Type":"ContainerStarted","Data":"3fb94a7564dcfedcf3f5b0be0a38571afdde9dc9ea70af30470cb6d70f06194b"} Nov 03 16:11:10.997483 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:10.997372 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfllm" event={"ID":"234cc11d-353c-4f0c-ab3b-4616ddd0b26d","Type":"ContainerStarted","Data":"615abb43cd5eb58c8a664f465fa2b1d75c5f0cdcb523b926fcee132c2f086b72"} Nov 03 16:11:10.998500 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:10.998483 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" event={"ID":"5e9344a7-9120-40ef-8671-5bc1fd4de206","Type":"ContainerStarted","Data":"f0cc589b8a988fb2e7c922b797125feaaa9fe29d96140442045106196f587069"} Nov 03 16:11:11.124417 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.124362 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" podStartSLOduration=3.019453454 podStartE2EDuration="5.124346689s" totalImagesPullingTime="2.104893235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:11.123538339 +0000 UTC m=+77.951648717" watchObservedRunningTime="2026-11-03 16:11:11.124346689 +0000 UTC m=+77.952457080" Nov 03 16:11:11.249026 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.248931 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hfllm" podStartSLOduration=2.141661318 podStartE2EDuration="4.248917878s" totalImagesPullingTime="2.029025427s" totalInitContainerRuntime="78.231133ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:11.247954646 +0000 UTC m=+78.076065023" watchObservedRunningTime="2026-11-03 16:11:11.248917878 +0000 UTC m=+78.077028277" Nov 03 16:11:11.408947 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.408915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:11.621682 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.621605 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:11.644655 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.644628 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nw79m"] Nov 03 16:11:11.646637 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.646620 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:11.703186 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.703158 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 16:11:11.703355 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.703227 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t4rpz\"" Nov 03 16:11:11.714650 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.714626 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nw79m"] Nov 03 16:11:11.749715 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.749679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cedb2971-5861-4832-bda4-242c206a0822-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:11.749864 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.749756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5h28\" (UniqueName: \"kubernetes.io/projected/cedb2971-5861-4832-bda4-242c206a0822-kube-api-access-r5h28\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:11.818077 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.818045 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:11.850903 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.850878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h28\" (UniqueName: \"kubernetes.io/projected/cedb2971-5861-4832-bda4-242c206a0822-kube-api-access-r5h28\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:11.851027 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.850933 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cedb2971-5861-4832-bda4-242c206a0822-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:11.853413 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.853370 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cedb2971-5861-4832-bda4-242c206a0822-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:11.912293 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.912230 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h28\" (UniqueName: \"kubernetes.io/projected/cedb2971-5861-4832-bda4-242c206a0822-kube-api-access-r5h28\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:11.957609 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:11.957575 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:12.011451 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.011363 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:12.076346 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.076315 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d7dfbc995-gq2ct"] Nov 03 16:11:12.079243 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.079225 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.136179 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.136148 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 16:11:12.136179 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.136169 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 16:11:12.136365 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.136169 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 16:11:12.136365 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.136340 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-94sd2\"" Nov 03 16:11:12.136365 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.136348 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 16:11:12.136446 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.136362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eql3qhcmfmus4\"" Nov 03 16:11:12.144061 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.144039 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d7dfbc995-gq2ct"] Nov 03 16:11:12.153978 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.153948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-client-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.154075 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.153991 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-metrics-server-audit-profiles\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.154075 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.154020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-client-certs\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.154149 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.154088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4nn\" (UniqueName: \"kubernetes.io/projected/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-kube-api-access-rx4nn\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.154149 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.154116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-tls\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.154263 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.154245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-audit-log\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.154317 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.154303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.255509 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.255475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-client-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.255674 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.255522 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-metrics-server-audit-profiles\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.255674 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.255544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-client-certs\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.255793 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.255670 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4nn\" (UniqueName: \"kubernetes.io/projected/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-kube-api-access-rx4nn\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.255793 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.255730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-tls\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.255882 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.255797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-audit-log\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.255882 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.255856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.256480 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.256343 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-audit-log\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.256642 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.256556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-metrics-server-audit-profiles\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.256722 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.256698 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.258429 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.258408 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-tls\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.258516 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.258461 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-client-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.258640 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.258620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-client-certs\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.285587 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.285530 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:12.317527 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.317505 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4nn\" (UniqueName: \"kubernetes.io/projected/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-kube-api-access-rx4nn\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.337433 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.337407 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nw79m"] Nov 03 16:11:12.338885 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:12.338860 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedb2971_5861_4832_bda4_242c206a0822.slice/crio-9242237ffc40dc6826d7f72c316274ea50e9b61bc516b42ee1341913f7100716 WatchSource:0}: Error finding container 9242237ffc40dc6826d7f72c316274ea50e9b61bc516b42ee1341913f7100716: Status 404 returned error can't find the container with id 9242237ffc40dc6826d7f72c316274ea50e9b61bc516b42ee1341913f7100716 Nov 03 16:11:12.389755 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.389725 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:12.475625 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.475593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:12.813825 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:12.813802 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d7dfbc995-gq2ct"] Nov 03 16:11:12.814980 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:12.814958 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8ec6fc_a61e_4d85_b9c8_f594647b0a96.slice/crio-9d437c224f8ae2c855cb246829fe1bffbebe3148e6e688cd8610a4e2a17b6fc3 WatchSource:0}: Error finding container 9d437c224f8ae2c855cb246829fe1bffbebe3148e6e688cd8610a4e2a17b6fc3: Status 404 returned error can't find the container with id 9d437c224f8ae2c855cb246829fe1bffbebe3148e6e688cd8610a4e2a17b6fc3 Nov 03 16:11:13.007341 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:13.007308 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" event={"ID":"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96","Type":"ContainerStarted","Data":"9d437c224f8ae2c855cb246829fe1bffbebe3148e6e688cd8610a4e2a17b6fc3"} Nov 03 16:11:13.008558 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:13.008528 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" event={"ID":"cedb2971-5861-4832-bda4-242c206a0822","Type":"ContainerStarted","Data":"9242237ffc40dc6826d7f72c316274ea50e9b61bc516b42ee1341913f7100716"} Nov 03 16:11:15.017343 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:15.017272 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" event={"ID":"cedb2971-5861-4832-bda4-242c206a0822","Type":"ContainerStarted","Data":"042c78eadf0e0247e27a01ba2697d621f60498b619a644bacda7105b2687ad1a"} Nov 03 16:11:15.017806 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:15.017537 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:15.023356 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:15.023331 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:11:15.149518 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:15.149448 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" podStartSLOduration=2.155415834 podStartE2EDuration="4.149429704s" totalImagesPullingTime="1.99401387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:15.149186312 +0000 UTC m=+81.977296689" watchObservedRunningTime="2026-11-03 16:11:15.149429704 +0000 UTC m=+81.977540083" Nov 03 16:11:15.966531 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:15.966496 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xg99b" Nov 03 16:11:16.023003 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.022972 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" event={"ID":"5e9344a7-9120-40ef-8671-5bc1fd4de206","Type":"ContainerStarted","Data":"dd2957dd94cf523dca91d32483b00a339346f5ab2b63fc87f00a0a4735b11a65"} Nov 03 16:11:16.023381 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.023037 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" event={"ID":"5e9344a7-9120-40ef-8671-5bc1fd4de206","Type":"ContainerStarted","Data":"c434e815e326c0c9cc479948d75cfcc6ad56f7c555ee8586cef6045ec1480d4a"} Nov 03 16:11:16.023381 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.023067 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" event={"ID":"5e9344a7-9120-40ef-8671-5bc1fd4de206","Type":"ContainerStarted","Data":"918aea7e7b464e585abe5a9607ea3ec574571471af61a3c5f7aee196dbcea291"} Nov 03 16:11:16.112996 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.112960 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:16.323547 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.323514 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:16.385169 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.385126 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:11:16.389030 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.389008 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.445396 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445359 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 16:11:16.445533 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445513 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 16:11:16.445533 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445522 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 16:11:16.445768 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445741 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 16:11:16.445835 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445778 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 16:11:16.445835 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445798 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2xgnz\"" Nov 03 16:11:16.445835 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445822 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 16:11:16.445934 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445877 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 16:11:16.445934 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.445926 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 16:11:16.446904 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.446890 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 16:11:16.446956 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.446893 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 16:11:16.490763 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.490736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-web-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.490911 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.490770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.490911 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.490794 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.490911 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.490855 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.490911 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.490878 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491034 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.490964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491034 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.490984 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491034 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491034 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config-out\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491144 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491082 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491180 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491230 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491261 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491236 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491289 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491325 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491353 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491340 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491380 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491357 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491408 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxqq\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-kube-api-access-8xxqq\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491440 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491432 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.491468 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.491451 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.498853 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.498808 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:11:16.532059 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.532034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:16.532284 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.532110 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 16:11:16.532284 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.532111 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7447pite6ult8\"" Nov 03 16:11:16.546777 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.546757 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 16:11:16.591912 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.591831 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-web-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.591912 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.591882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.591912 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.591909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.591945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.591969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592048 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592161 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config-out\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592406 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592435 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592833 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592458 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxqq\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-kube-api-access-8xxqq\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592833 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592507 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.592833 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.592531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.593573 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.593733 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.594576 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.594602 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.595082 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.595316 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.595381 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.596064 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.595789 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.597618 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.597144 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.597618 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.597565 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.597618 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.597574 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-web-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.597925 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.597896 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config-out\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.598151 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.598127 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.598714 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.598692 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.599126 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.599108 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.599247 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.599187 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.599602 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.599581 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.602581 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.602554 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 16:11:16.602946 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.602930 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 16:11:16.609447 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.608122 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.613850 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.613829 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.659519 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.659475 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxqq\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-kube-api-access-8xxqq\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.715629 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.715598 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:16.963634 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:16.963602 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:17.029275 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:17.029247 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" event={"ID":"5e9344a7-9120-40ef-8671-5bc1fd4de206","Type":"ContainerStarted","Data":"af59e0b779939327ee662a56e786bdaefcc1ee5b15fc96cef14fcd978480774f"} Nov 03 16:11:17.029600 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:17.029348 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" event={"ID":"5e9344a7-9120-40ef-8671-5bc1fd4de206","Type":"ContainerStarted","Data":"acd59860c9d777493ffc70dcee8f2114b622fb457b15cbea466cbdc1a107d063"} Nov 03 16:11:17.031051 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:17.031026 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" event={"ID":"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96","Type":"ContainerStarted","Data":"2f77767ef2d12eae6bdd805551945779deeb3bed450b5c5de990e4fa273bf7cc"} Nov 03 16:11:17.162008 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:17.161910 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" podStartSLOduration=1.94937195 podStartE2EDuration="5.161897572s" totalImagesPullingTime="3.212525622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:17.161477116 +0000 UTC m=+83.989587494" watchObservedRunningTime="2026-11-03 16:11:17.161897572 +0000 UTC m=+83.990007948" Nov 03 16:11:17.193589 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:17.193547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:17.217296 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:17.217266 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:11:17.268745 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:11:17.268716 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29f1414_146f_4bb2_a62c_a29b6f54ff75.slice/crio-b32d83ba628570afdfca0d09fb9053490d77ac12f9f272b573b2cf393b32481c WatchSource:0}: Error finding container b32d83ba628570afdfca0d09fb9053490d77ac12f9f272b573b2cf393b32481c: Status 404 returned error can't find the container with id b32d83ba628570afdfca0d09fb9053490d77ac12f9f272b573b2cf393b32481c Nov 03 16:11:18.035138 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:18.035099 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerStarted","Data":"b32d83ba628570afdfca0d09fb9053490d77ac12f9f272b573b2cf393b32481c"} Nov 03 16:11:18.038270 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:18.038240 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" event={"ID":"5e9344a7-9120-40ef-8671-5bc1fd4de206","Type":"ContainerStarted","Data":"4c1bc447c07794777059b7a9f029ecbcdd9b9a8ab1f62b856f62c89cc8f166ee"} Nov 03 16:11:18.169767 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:18.169713 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" podStartSLOduration=2.877142717 podStartE2EDuration="9.169697115s" totalImagesPullingTime="6.292554398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:18.168354497 +0000 UTC m=+84.996464874" watchObservedRunningTime="2026-11-03 16:11:18.169697115 +0000 UTC m=+84.997807491" Nov 03 16:11:18.330396 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:18.330256 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:18.539088 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:18.539042 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:19.043441 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:19.043411 2572 generic.go:356] "Generic (PLEG): container finished" podID="e29f1414-146f-4bb2-a62c-a29b6f54ff75" containerID="7ba7cfe37605f154077c556bc745f887059ae5bf0d3e847100123354654f7e0f" exitCode=0 Nov 03 16:11:19.043872 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:19.043504 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerDied","Data":"7ba7cfe37605f154077c556bc745f887059ae5bf0d3e847100123354654f7e0f"} Nov 03 16:11:19.043872 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:19.043690 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:20.054955 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:20.054928 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:11:23.068379 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.068345 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerStarted","Data":"6fe918c48088dcef280ec891ecdf1beb372a253a411714c7bde80cecfef513e5"} Nov 03 16:11:23.068751 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.068412 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerStarted","Data":"e0c560428cd16f8ec51438b61f422937b8929dc9415e109e5be91678a7579d3a"} Nov 03 16:11:23.068751 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.068439 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerStarted","Data":"8a53e13da046599dbd95aa1b5aaf5cfce29b5aaa1cc8e2becbf3332b11075868"} Nov 03 16:11:23.068751 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.068470 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerStarted","Data":"dd8215038fc28e99aecdf5488b460ad12029341a9a0f906bd87edc6b12789cdd"} Nov 03 16:11:23.068751 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.068494 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerStarted","Data":"fca3f95279206e213d747a69f8f4cb517df2763338dd0d7aeb3955c271143377"} Nov 03 16:11:23.068751 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.068519 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e29f1414-146f-4bb2-a62c-a29b6f54ff75","Type":"ContainerStarted","Data":"cf39782b3d6d5eadff6edf2935081c12ef9bdc77f9e0986643c5e43a65a4dfb3"} Nov 03 16:11:23.243684 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.243627 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.355136 podStartE2EDuration="7.243614471s" totalImagesPullingTime="4.871724161s" totalInitContainerRuntime="16.75431ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:23.241922106 +0000 UTC m=+90.070032484" watchObservedRunningTime="2026-11-03 16:11:23.243614471 +0000 UTC m=+90.071724880" Nov 03 16:11:23.613040 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.613011 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:23.880001 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:23.879924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:26.580244 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:26.580191 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:26.716290 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:26.716254 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:11:26.774990 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:26.774959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:27.873402 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:27.873359 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:28.076693 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:28.076663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:31.492573 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:11:31.492537 2572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:11:32.390166 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:32.390133 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:32.390390 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:32.390239 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:41.628623 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:41.628590 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:41.821767 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:41.821734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:46.014294 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:46.014191 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:46.204627 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:46.204586 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:52.395991 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:52.395962 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:11:52.399593 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:11:52.399571 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:01.492778 ip-10-0-135-216 kubenswrapper[2572]: E1103 16:12:01.492739 2572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:12:03.613508 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.613457 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:12:03.613950 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.613536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:03.613950 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.613579 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:12:03.667336 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.667304 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:12:03.719564 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.719531 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:12:03.719564 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.719552 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:12:03.726954 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.726919 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:03.727106 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.726933 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:12:03.727290 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.727276 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:12:03.736566 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.736537 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:12:03.879486 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.879405 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rqr4v\"" Nov 03 16:12:03.885877 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.885849 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:12:03.886166 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.886147 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7zlt\"" Nov 03 16:12:03.890492 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.890250 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zsc2l\"" Nov 03 16:12:03.893348 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.893330 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:03.897095 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:03.897074 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:12:04.272585 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:04.272565 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fddzx"] Nov 03 16:12:04.276248 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:12:04.275361 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8540df11_165c_4b01_b8a3_4042a8f6fdab.slice/crio-4452adf36900b0b1cd18a3c51a374294c9200a6fb22f2f2f20a3c63e81c79e5a WatchSource:0}: Error finding container 4452adf36900b0b1cd18a3c51a374294c9200a6fb22f2f2f20a3c63e81c79e5a: Status 404 returned error can't find the container with id 4452adf36900b0b1cd18a3c51a374294c9200a6fb22f2f2f20a3c63e81c79e5a Nov 03 16:12:04.324958 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:04.324932 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5dbjc"] Nov 03 16:12:04.326282 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:04.326255 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67tfw"] Nov 03 16:12:04.326699 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:12:04.326670 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf14ab9_d061_4cfb_bc58_68ecf17a10a7.slice/crio-9ebd1669d012c917e5f5a5f98e924ff4bb3b45356e03383dd033b4686e415e1c WatchSource:0}: Error finding container 9ebd1669d012c917e5f5a5f98e924ff4bb3b45356e03383dd033b4686e415e1c: Status 404 returned error can't find the container with id 9ebd1669d012c917e5f5a5f98e924ff4bb3b45356e03383dd033b4686e415e1c Nov 03 16:12:04.327119 ip-10-0-135-216 kubenswrapper[2572]: W1103 16:12:04.327095 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab478348_e9ee_4b58_a398_422dd7e03f0b.slice/crio-83749d3087fca32ebacf984bd6d694d91fac851df15604add05aff0a98ab76ca WatchSource:0}: Error finding container 83749d3087fca32ebacf984bd6d694d91fac851df15604add05aff0a98ab76ca: Status 404 returned error can't find the container with id 83749d3087fca32ebacf984bd6d694d91fac851df15604add05aff0a98ab76ca Nov 03 16:12:05.209300 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:05.209254 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-67tfw" event={"ID":"caf14ab9-d061-4cfb-bc58-68ecf17a10a7","Type":"ContainerStarted","Data":"9ebd1669d012c917e5f5a5f98e924ff4bb3b45356e03383dd033b4686e415e1c"} Nov 03 16:12:05.210519 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:05.210495 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5dbjc" event={"ID":"ab478348-e9ee-4b58-a398-422dd7e03f0b","Type":"ContainerStarted","Data":"83749d3087fca32ebacf984bd6d694d91fac851df15604add05aff0a98ab76ca"} Nov 03 16:12:05.211568 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:05.211548 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fddzx" event={"ID":"8540df11-165c-4b01-b8a3-4042a8f6fdab","Type":"ContainerStarted","Data":"4452adf36900b0b1cd18a3c51a374294c9200a6fb22f2f2f20a3c63e81c79e5a"} Nov 03 16:12:06.928019 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:06.927987 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:07.120351 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:07.120309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:07.224362 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:07.224279 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fddzx" event={"ID":"8540df11-165c-4b01-b8a3-4042a8f6fdab","Type":"ContainerStarted","Data":"9d69e1c2a3141e6fc1bc072c5291cff62b1f3eb77057966a5d12927f01818e87"} Nov 03 16:12:07.224362 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:07.224348 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fddzx" event={"ID":"8540df11-165c-4b01-b8a3-4042a8f6fdab","Type":"ContainerStarted","Data":"deac354221d28bf971f5e731541c52e2fd07e5eec69dc0ab7f159fb5576b2abe"} Nov 03 16:12:07.347514 ip-10-0-135-216 kubenswrapper[2572]: I1103 16:12:07.347458 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fddzx" podStartSLOduration=131.397600837 podStartE2EDuration="2m13.347440143s" totalImagesPullingTime="1.949839306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:07.346480893 +0000 UTC m=+134.174591273" watchObservedRunningTime="2026-11-03 16:12:07.347440143 +0000 UTC m=+134.175550520" Nov 03 16:12:09.796726 ip-10-0-135-216 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 16:12:09.817670 ip-10-0-135-216 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 16:12:09.817840 ip-10-0-135-216 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 16:12:09.818013 ip-10-0-135-216 systemd[1]: kubelet.service: Consumed 6.589s CPU time. Nov 03 16:12:09.836014 ip-10-0-135-216 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:12:09.910073 ip-10-0-135-216 kubenswrapper[5946]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:12:09.910073 ip-10-0-135-216 kubenswrapper[5946]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:12:09.910073 ip-10-0-135-216 kubenswrapper[5946]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:12:09.910073 ip-10-0-135-216 kubenswrapper[5946]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:12:09.912114 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912028 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.912114 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912115 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912119 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912123 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912126 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912129 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912133 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912137 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912140 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912144 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912147 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912150 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912154 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912158 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912161 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912164 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912167 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912170 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912173 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912175 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.912199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912178 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912181 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912183 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912186 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912189 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912192 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912195 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912197 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912218 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912222 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912225 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912228 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912230 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912233 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912236 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912238 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912241 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912243 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912246 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912249 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.912704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912252 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912255 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912257 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912260 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912262 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912265 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912268 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912270 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912273 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912276 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912278 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912281 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912284 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912286 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912289 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912291 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912294 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912297 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912299 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912302 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.913192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912304 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912307 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912309 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912311 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912314 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912316 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912319 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912321 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912324 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912326 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912328 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912331 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912334 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912336 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912339 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912341 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912343 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912346 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912349 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912351 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.913704 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912353 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912356 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912359 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912361 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912364 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912366 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912369 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912371 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912374 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912376 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912379 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912381 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912384 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912386 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912389 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912391 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912394 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912396 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912399 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912404 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.914216 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912408 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912412 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912415 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912418 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912420 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.912424 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913186 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913193 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913197 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913227 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913233 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913238 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913242 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913246 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913251 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913254 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913257 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913260 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913262 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.914728 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913265 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913267 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913270 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913272 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913275 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913277 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913280 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913282 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913285 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913288 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913290 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913293 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913295 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913298 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913300 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913303 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913305 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913308 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913310 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913321 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.915198 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913324 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913326 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913329 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913331 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913334 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913336 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913339 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913342 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913345 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913348 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913350 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913353 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913355 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913357 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913360 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913363 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913366 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913368 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913371 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913373 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.915725 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913376 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913378 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913381 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913383 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913386 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913388 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913393 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913396 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913399 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913402 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913404 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913407 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913411 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913413 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913416 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913418 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913421 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913423 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913425 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.916274 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913428 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913431 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913433 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913436 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913438 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913441 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913443 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913446 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913448 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913451 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913453 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913456 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913458 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913461 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913463 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913466 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913468 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913470 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913473 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913475 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.916740 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913477 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913480 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913482 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913485 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913490 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913495 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913499 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913501 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913504 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913515 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913518 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913520 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913523 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913526 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.913999 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914004 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914007 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914009 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914012 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.917231 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914015 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914017 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914020 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914023 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914026 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914029 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914031 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914034 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914037 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914039 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914042 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914045 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914047 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914050 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914053 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914056 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914058 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914061 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914064 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914067 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.917693 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914070 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914072 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914074 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914077 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914079 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914081 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914084 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914087 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914089 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914091 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914094 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914096 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914098 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914101 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914104 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914106 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914108 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914111 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914113 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914116 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.918434 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914119 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914121 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914125 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914127 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914130 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914132 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914135 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914137 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914140 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914142 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914145 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914147 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914150 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914153 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914155 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914157 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914160 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914162 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914165 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914167 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.919320 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914170 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914172 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914175 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914177 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914180 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914183 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914187 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914190 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914193 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914197 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914199 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914223 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914228 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914233 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914236 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914239 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914242 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914244 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914247 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.920194 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914249 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914252 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914254 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914257 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914260 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914262 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914265 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914268 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914271 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914274 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914277 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914279 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914282 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914284 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914287 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914289 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914292 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914294 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914297 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914299 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.921012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914303 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.914305 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914419 5946 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914431 5946 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914437 5946 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914442 5946 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914446 5946 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914449 5946 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914454 5946 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914458 5946 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914461 5946 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914464 5946 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914468 5946 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914471 5946 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914474 5946 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914477 5946 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914480 5946 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914483 5946 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914488 5946 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914491 5946 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914495 5946 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914498 5946 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914501 5946 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914504 5946 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:12:09.921921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914507 5946 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914511 5946 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914514 5946 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914518 5946 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914521 5946 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914524 5946 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914527 5946 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914530 5946 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914533 5946 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914536 5946 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914540 5946 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914543 5946 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914546 5946 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914549 5946 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914552 5946 flags.go:64] FLAG: --enable-server="true" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914555 5946 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914560 5946 flags.go:64] FLAG: --event-burst="100" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914563 5946 flags.go:64] FLAG: --event-qps="50" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914566 5946 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914569 5946 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914572 5946 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914577 5946 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914580 5946 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914583 5946 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914586 5946 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:12:09.922660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914589 5946 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914594 5946 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914597 5946 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914601 5946 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914604 5946 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914607 5946 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914610 5946 flags.go:64] FLAG: --feature-gates="" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914614 5946 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914618 5946 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914621 5946 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914624 5946 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914627 5946 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914630 5946 flags.go:64] FLAG: --help="false" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914633 5946 flags.go:64] FLAG: --hostname-override="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914636 5946 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914640 5946 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914643 5946 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914646 5946 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914650 5946 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914653 5946 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914658 5946 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914661 5946 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914664 5946 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:12:09.923405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914667 5946 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914670 5946 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914673 5946 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914676 5946 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914678 5946 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914682 5946 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914684 5946 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914687 5946 flags.go:64] FLAG: --lock-file="" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914690 5946 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914693 5946 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914697 5946 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914702 5946 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914705 5946 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914709 5946 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914712 5946 flags.go:64] FLAG: --logging-format="text" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914715 5946 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914718 5946 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914721 5946 flags.go:64] FLAG: --manifest-url="" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914724 5946 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914728 5946 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914732 5946 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914736 5946 flags.go:64] FLAG: --max-pods="110" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914739 5946 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914742 5946 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914745 5946 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914748 5946 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:12:09.923970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914751 5946 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914754 5946 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914757 5946 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914765 5946 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914768 5946 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914771 5946 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914775 5946 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914777 5946 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914780 5946 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914783 5946 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914786 5946 flags.go:64] FLAG: --port="10250" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914789 5946 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914793 5946 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-05e91847f7ec0d696" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914796 5946 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914799 5946 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914802 5946 flags.go:64] FLAG: --register-node="true" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914806 5946 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914811 5946 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914814 5946 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914817 5946 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914820 5946 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914824 5946 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914827 5946 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914834 5946 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914837 5946 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914840 5946 flags.go:64] FLAG: --runonce="false" Nov 03 16:12:09.924646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914843 5946 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914846 5946 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914849 5946 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914852 5946 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914855 5946 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914858 5946 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914861 5946 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914864 5946 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914867 5946 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914873 5946 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914876 5946 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914879 5946 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914882 5946 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914885 5946 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914888 5946 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914894 5946 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914897 5946 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914900 5946 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914903 5946 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914906 5946 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914909 5946 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914912 5946 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914917 5946 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914920 5946 flags.go:64] FLAG: --v="2" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914924 5946 flags.go:64] FLAG: --version="false" Nov 03 16:12:09.925376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914929 5946 flags.go:64] FLAG: --vmodule="" Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914933 5946 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.914936 5946 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915071 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915076 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915078 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915081 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915084 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915087 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915089 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915092 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915095 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915097 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915100 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915102 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915105 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915109 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915111 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915114 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.926006 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915116 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915119 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915122 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915124 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915126 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915129 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915131 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915134 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915137 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915139 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915144 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915146 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915148 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915151 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915153 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915156 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915159 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915161 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915164 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915166 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.926700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915169 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915171 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915174 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915176 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915180 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915184 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915187 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915190 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915192 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915196 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915199 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915219 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915222 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915225 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915228 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915231 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915233 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915236 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915238 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915241 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.927282 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915243 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915246 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915249 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915252 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915254 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915257 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915259 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915262 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915264 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915267 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915270 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915273 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915275 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915277 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915280 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915283 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915285 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915288 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915290 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915293 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.928067 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915295 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915299 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915302 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915304 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915307 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915309 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915312 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915315 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915317 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915320 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915322 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915325 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915327 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915330 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915332 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915335 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915337 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915340 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915344 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915348 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.928706 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915351 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915354 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915357 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915360 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915363 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915365 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915367 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915370 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915372 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915375 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.915401 5946 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915436 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915441 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915444 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915449 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.929251 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915451 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915454 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915457 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915460 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915462 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915465 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915467 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915470 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915473 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915475 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915478 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915480 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915483 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915485 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915488 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915490 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915493 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915496 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915498 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.929836 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915501 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915504 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915507 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915509 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915512 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915514 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915517 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915520 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915522 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915525 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915527 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915530 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915533 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915536 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915539 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915543 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915547 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915550 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915552 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915555 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.930571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915558 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915560 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915563 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915566 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915568 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915571 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915574 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915576 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915578 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915581 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915583 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915586 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915588 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915591 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915593 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915596 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915598 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915601 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915603 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915605 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.931445 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915608 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915611 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915614 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915616 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915620 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915623 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915625 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915628 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915630 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915633 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915635 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915638 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915640 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915643 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915645 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915648 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915650 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915653 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915655 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915657 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.932457 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915660 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915662 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915665 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915668 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915671 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915673 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915676 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915678 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915681 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915685 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915688 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915691 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915694 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915698 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915701 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915703 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915707 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915710 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915712 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.933286 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915715 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915717 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915720 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.915722 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.922084 5946 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.922101 5946 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.922129 5946 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.922135 5946 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922196 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922218 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922224 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922228 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922233 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922236 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922240 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922243 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922247 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922249 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922252 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922255 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.934115 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922257 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922260 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922262 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922268 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922271 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922275 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922278 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922281 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922283 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922286 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922289 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922292 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922295 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922297 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922300 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922303 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922306 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922308 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922311 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.934988 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922314 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922316 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922319 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922321 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922324 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922326 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922329 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922331 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922334 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922336 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922338 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922341 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922343 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922346 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922348 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922352 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922355 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922357 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922360 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.935797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922363 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922366 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922368 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922371 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922374 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922376 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922379 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922382 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922385 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922387 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922390 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922392 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922395 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922398 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922400 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922403 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922405 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922408 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922411 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922415 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.936618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922419 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922422 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922425 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922428 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922430 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922433 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922436 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922439 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922441 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922444 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922447 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922450 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922453 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922455 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922458 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922460 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922463 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922465 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922468 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922470 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.937601 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922472 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922475 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922478 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922481 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922483 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922485 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922488 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922490 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922493 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922495 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922498 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922500 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922503 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922505 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922507 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922510 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.938523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.922540 5946 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922686 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922691 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922693 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922696 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922699 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922702 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922705 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922708 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922712 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922715 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922718 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922720 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922723 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922725 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922728 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922730 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922733 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922735 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922738 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.939249 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922740 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922743 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922745 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922748 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922750 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922753 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922755 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922757 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922760 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922763 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922765 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922768 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922770 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922773 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922775 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922778 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922780 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922783 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922786 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.940185 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922789 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922791 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922794 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922796 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922798 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922801 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922803 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922806 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922808 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922811 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922813 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922815 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922818 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922821 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922823 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922825 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922828 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922830 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922833 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922835 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.941137 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922837 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922840 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922843 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922845 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922847 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922850 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922852 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922855 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922857 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922860 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922863 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922865 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922868 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922871 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922874 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922876 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922879 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922881 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922884 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922886 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.942134 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922888 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922891 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922893 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922896 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922900 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922903 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922906 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922909 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922911 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922914 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922916 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922918 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922921 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922924 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922927 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922930 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922933 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922935 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922938 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.942964 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922941 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922943 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922946 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922948 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922951 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922953 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922956 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922959 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.922961 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.922984 5946 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923016 5946 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923020 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923023 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923026 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923028 5946 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923031 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:12:09.943659 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923034 5946 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923036 5946 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923039 5946 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923042 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923045 5946 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923047 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923050 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923052 5946 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923055 5946 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923057 5946 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923060 5946 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923063 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923065 5946 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923068 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923070 5946 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923073 5946 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923075 5946 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923078 5946 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923080 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:12:09.944064 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923082 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923086 5946 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923092 5946 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923095 5946 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923097 5946 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923100 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923103 5946 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923106 5946 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923108 5946 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923111 5946 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923113 5946 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923116 5946 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923118 5946 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923121 5946 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923123 5946 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923125 5946 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923128 5946 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923131 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923133 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923135 5946 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:12:09.944554 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923138 5946 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923140 5946 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923143 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923146 5946 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923149 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923151 5946 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923154 5946 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923156 5946 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923159 5946 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923162 5946 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923164 5946 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923167 5946 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923169 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923172 5946 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923175 5946 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923178 5946 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923180 5946 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923183 5946 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923186 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923188 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:12:09.945050 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923191 5946 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923193 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923196 5946 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923198 5946 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923222 5946 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923226 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923229 5946 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923231 5946 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923234 5946 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923236 5946 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923238 5946 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923241 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923243 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923246 5946 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923249 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923252 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923255 5946 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923257 5946 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923259 5946 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923262 5946 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:12:09.945576 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923264 5946 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923267 5946 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923269 5946 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923272 5946 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923275 5946 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923277 5946 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923281 5946 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923283 5946 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923286 5946 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923289 5946 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923291 5946 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923295 5946 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923300 5946 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923304 5946 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923309 5946 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923317 5946 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923323 5946 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923328 5946 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923332 5946 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:12:09.946074 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923335 5946 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:12:09.923338 5946 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.923398 5946 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.924382 5946 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.924437 5946 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.924618 5946 server.go:1014] "Starting client certificate rotation" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.924767 5946 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.924822 5946 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:04:54 +0000 UTC" deadline="2028-06-06 09:41:39.842848134 +0000 UTC" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.924849 5946 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13937h29m29.918001276s" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.925065 5946 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.926110 5946 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.927448 5946 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.929460 5946 log.go:25] "Validated CRI v1 image API" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.930183 5946 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:12:09.946578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.933384 5946 fs.go:135] Filesystem UUIDs: map[5980846a-5135-4947-b119-a271f82233d5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7aab4194-4be6-4a56-9482-f2648c83cfbc:/dev/nvme0n1p3] Nov 03 16:12:09.947166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.933411 5946 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0710ffa9a941cc18f494e9dd2af469e43af2875eda5f24677845d7ec0c82d9b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0710ffa9a941cc18f494e9dd2af469e43af2875eda5f24677845d7ec0c82d9b2/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27547d86ebff3bbf4b258bacccf495303f620bfae8ba86730882b7bde3f8e921/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27547d86ebff3bbf4b258bacccf495303f620bfae8ba86730882b7bde3f8e921/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28b1f276c9a8cc016ccc7119edebc5f02f396ad024aac0fdad27a3009cf07301/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28b1f276c9a8cc016ccc7119edebc5f02f396ad024aac0fdad27a3009cf07301/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3940a1ec9ee29b1dbf2f584e8b94b030a938a43b492a0bf8996d34f977202cb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3940a1ec9ee29b1dbf2f584e8b94b030a938a43b492a0bf8996d34f977202cb6/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4452adf36900b0b1cd18a3c51a374294c9200a6fb22f2f2f20a3c63e81c79e5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4452adf36900b0b1cd18a3c51a374294c9200a6fb22f2f2f20a3c63e81c79e5a/userdata/shm major:0 minor:431 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/446f06d2e92ad81b4ec59682f6b21398563e75b42d49440ea4b7342652856a0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/446f06d2e92ad81b4ec59682f6b21398563e75b42d49440ea4b7342652856a0b/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71b2223e985e4191587f83be652b00056d583876f94db1ba5ebc2b750823d2de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71b2223e985e4191587f83be652b00056d583876f94db1ba5ebc2b750823d2de/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a5bf72fdce77c3fe97c071e3a35ab348b54ec0f28dc84348b84a9b752a049ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a5bf72fdce77c3fe97c071e3a35ab348b54ec0f28dc84348b84a9b752a049ce/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83749d3087fca32ebacf984bd6d694d91fac851df15604add05aff0a98ab76ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83749d3087fca32ebacf984bd6d694d91fac851df15604add05aff0a98ab76ca/userdata/shm major:0 minor:433 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87c40c67d97f6d4f229fe3b64865f4d0bb39bf93f4f9519a8eea8f9fd8e302d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87c40c67d97f6d4f229fe3b64865f4d0bb39bf93f4f9519a8eea8f9fd8e302d8/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9242237ffc40dc6826d7f72c316274ea50e9b61bc516b42ee1341913f7100716/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9242237ffc40dc6826d7f72c316274ea50e9b61bc516b42ee1341913f7100716/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a7cc691cd382057664b54b18d4c13cf31888e63190e40eded9609f9e29f97c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a7cc691cd382057664b54b18d4c13cf31888e63190e40eded9609f9e29f97c7/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d437c224f8ae2c855cb246829fe1bffbebe3148e6e688cd8610a4e2a17b6fc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d437c224f8ae2c855cb246829fe1bffbebe3148e6e688cd8610a4e2a17b6fc3/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ebd1669d012c917e5f5a5f98e924ff4bb3b45356e03383dd033b4686e415e1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ebd1669d012c917e5f5a5f98e924ff4bb3b45356e03383dd033b4686e415e1c/userdata/shm major:0 minor:435 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a12181362c94a76b51bb5c5bcfe89314053884f6ab2141085f7ab9036a2a26be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a12181362c94a76b51bb5c5bcfe89314053884f6ab2141085f7ab9036a2a26be/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4d113c1a2471e4a44779b05c6a78480ce7c8147b02d073e989def2314140f2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4d113c1a2471e4a44779b05c6a78480ce7c8147b02d073e989def2314140f2c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae082eed5008916f48815573bdc06e2bb3521ac656ac4c5269ff2348815626c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae082eed5008916f48815573bdc06e2bb3521ac656ac4c5269ff2348815626c6/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b32d83ba628570afdfca0d09fb9053490d77ac12f9f272b573b2cf393b32481c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b32d83ba628570afdfca0d09fb9053490d77ac12f9f272b573b2cf393b32481c/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d28b27f98f82bd7dd24b18d0f8f41eec8a911886466d382660e839f75c09ba04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d28b27f98f82bd7dd24b18d0f8f41eec8a911886466d382660e839f75c09ba04/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebf950b28b0574653f84097e73ed1ec5c049bed29fa96063cf4289dded008f6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebf950b28b0574653f84097e73ed1ec5c049bed29fa96063cf4289dded008f6f/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0cc589b8a988fb2e7c922b797125feaaa9fe29d96140442045106196f587069/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0cc589b8a988fb2e7c922b797125feaaa9fe29d96140442045106196f587069/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe7ec78289e3c0d5c945b9ea4d3999bf62c7611201c21f4d95150d6516e442a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe7ec78289e3c0d5c945b9ea4d3999bf62c7611201c21f4d95150d6516e442a7/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05b3881e-7078-461d-9c94-69d049ccc417/volumes/kubernetes.io~projected/kube-api-access-8pxqv:{mountpoint:/var/lib/kubelet/pods/05b3881e-7078-461d-9c94-69d049ccc417/volumes/kubernetes.io~projected/kube-api-access-8pxqv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05b3881e-7078-461d-9c94-69d049ccc417/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/05b3881e-7078-461d-9c94-69d049ccc417/volumes/kubernetes.io~secret/cert major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2218179a-fa77-4676-ad59-885f522a5039/volumes/kubernetes.io~projected/kube-api-access-j62d8:{mountpoint:/var/lib/kubelet/pods/2218179a-fa77-4676-ad59-885f522a5039/volumes/kubernetes.io~projected/kube-api-access-j62d8 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2218179a-fa77-4676-ad59-885f522a5039/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/2218179a-fa77-4676-ad59-885f522a5039/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~projected/kube-api-access-lhtmh:{mountpoint:/var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~projected/kube-api-access-lhtmh major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/289d2a22-071a-436a-afa6-8bfecabaa60c/volumes/kubernetes.io~projected/kube-api-access-c56sh:{mountpoint:/var/lib/kubelet/pods/289d2a22-071a-436a-afa6-8bfecabaa60c/volumes/kubernetes.io~projected/kube-api-access-c56sh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~projected/kube-api-access-w5fdm:{mountpoint:/var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~projected/kube-api-access-w5fdm major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volumes/kubernetes.io~projected/kube-api-access-qwd77:{mountpoint:/var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volumes/kubernetes.io~projected/kube-api-access-qwd77 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~projected/kube-api-access-jn6md:{mountpoint:/var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~projected/kube-api-access-jn6md major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/507f7522-ec6f-4e48-b362-68e1c9dafeb8/volumes/kubernetes.io~projected/kube-api-access-jp64h:{mountpoint:/var/lib/kubelet/pods/507f7522-ec6f-4e48-b362-68e1c9dafeb8/volumes/kubernetes.io~projected/kube-api-access-jp64h major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~projected/kube-api-access-rx4nn:{mountpoint:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~projected/kube-api-access-rx4nn major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~projected/kube-api-access-7xrll:{mountpoint:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~projected/kube-api-access-7xrll major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~projected/kube-api-access-mvlq9:{mountpoint:/var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~projected/kube-api-access-mvlq9 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61daa065-4042-47db-add5-514189226920/volumes/kubernetes.io~projected/kube-api-access-skprg:{mountpoint:/var/lib/kubelet/pods/61daa065-4042-47db-add5-514189226920/volumes/kubernetes.io~projected/kube-api-access-skprg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7da5e2a9-8f58-4ee1-8a00-c443413a249c/volumes/kubernetes.io~projected/kube-api-access-8czjl:{mountpoint:/var/lib/kubelet/pods/7da5e2a9-8f58-4ee1-8a00-c443413a249c/volumes/kubernetes.io~projected/kube-api-access-8czjl major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7da5e2a9-8f58-4ee1-8a00-c443413a249c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7da5e2a9-8f58-4ee1-8a00-c443413a249c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:123 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8540df11-165c-4b01-b8a3-4042a8f6fdab/volumes/kubernetes.io~projected/kube-api-access-6fjnh:{mountpoint:/var/lib/kubelet/pods/8540df11-165c-4b01-b8a3-4042a8f6fdab/volumes/kubernetes.io~projected/kube-api-access-6fjnh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8540df11-165c-4b01-b8a3-4042a8f6fdab/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/8540df11-165c-4b01-b8a3-4042a8f6fdab/volumes/kubernetes.io~secret/metrics-certs major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9189a7df-aae8-4728-be2b-569f2cffebc3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9189a7df-aae8-4728-be2b-569f2cffebc3/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab478348-e9ee-4b58-a398-422dd7e03f0b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ab478348-e9ee-4b58-a398-422dd7e03f0b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3/volumes/kubernetes.io~projected/kube-api-access-7648n:{mountpoint:/var/lib/kubelet/pods/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3/volumes/kubernetes.io~projected/kube-api-access-7648n major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0c67fab-535b-4a8e-a336-ca475185c3dd/volumes/kubernetes.io~projected/kube-api-access-ltrdq:{mountpoint:/var/lib/kubelet/pods/c0c67fab-535b-4a8e-a336-ca475185c3dd/volumes/kubernetes.io~projected/kube-api-access-ltrdq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caf14ab9-d061-4cfb-bc58-68ecf17a10a7/volumes/kubernetes.io~projected/kube-api-access-42gvz:{mountpoint:/var/lib/kubelet/pods/caf14ab9-d061-4cfb-bc58-68ecf17a10a7/volumes/kubernetes.io~projected/kube-api-access-42gvz major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cedb2971-5861-4832-bda4-242c206a0822/volumes/kubernetes.io~projected/kube-api-access-r5h28:{mountpoint:/var/lib/kubelet/pods/cedb2971-5861-4832-bda4-242c206a0822/volumes/kubernetes.io~projected/kube-api-access-r5h28 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cedb2971-5861-4832-bda4-242c206a0822/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/cedb2971-5861-4832-bda4-242c206a0822/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/prometheus/15 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~empty-dir/config-out major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~projected/kube-api-access-8xxqq:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~projected/kube-api-access-8xxqq major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~projected/tls-assets major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/config major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/web-config major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecdd9d33-7747-417b-bff0-688639640a48/volumes/kubernetes.io~projected/kube-api-access-kr6q9:{mountpoint:/var/lib/kubelet/pods/ecdd9d33-7747-417b-bff0-688639640a48/volumes/kubernetes.io~projected/kube-api-access-kr6q9 major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8dee87d1f20f326053a180c05ea2baf51e2cde1b0b81710904d7a3e4fe4e544c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/19bef42bffd3fdd654197b8fbb43a47ac729393729ef97860b182630aa39e9c6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/44a413fb47e1a2ff20b704c1bfb0fe51126d822c2631f27395467bae364c97dc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1ffdc8695b7278fc48f9756955b303cb55ffac442c43446efe6b5c48195dfc08/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/e9a70bd7de86e5224a23dd7eaecc8ee8aef661369bc588921f6278b080f296ac/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-117:{mountpoint:/var/lib/containers/storage/overlay/6c0936a5235f0d047e296291762de541ac94c9662e1ddcf6cd911a2695123379/merged major:0 minor:117 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/63275edfefe69734e0418bdeb00bf29fb4afa04d2096789ab3c173e55e9a5024/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b9e9d651a56e2937797a8ffbc89deb29d9becb75d4af8b1a0fc2f46c28c49603/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/8e79b5de0351e4a752aeff06bb7a057cbabbe2d4c5162593d3fd6d0d2eff92d2/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/12081fba6c71193bce0e8aacecbdf5feb968dc92647a8165a80a276f95525535/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/33a39233796e698d3cc05ffd5048097915625fa7c96d30f07e4afcb28ee19c05/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/5652da91b14aeb3745576186bb2e6fc120a0735908607787a7b1e5b9af83c6b1/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ad5994545d2e06b639af049603e9ef7c2c546cdc9a6648b388cfa7c740fbbd12/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/a74f321a29eb41bf9bf524a963b117fa06bdf39106aa6498fb29141b47e4338d/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/d5d5e9198277ab0cecf257329d7ad7b2234f3fc0d07da681a161387683e7a716/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/67b1c9f9fd22f80891629710d4ff3ea2284ae4fd2d284add94b5070c003179bc/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/7fc1a746c1ac82f6bc7b7080f19ddff1067c20fcd9bc753e25014e652918c283/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/7cf21cf26a9aeeb14ad638f3fb3092220df63d68f11c867fc8420db0da2bd7fb/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/858f435dd3023b078b128d50bb53de2b74a7a7369301a1cd93793e3ab97fde8b/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/9078e896b3aa6c72e484e0011d0bce7efd838733b40cefc9716fb71e8c5ff7c0/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b1dafce9ad4cd240fab72230965a84e9a668a48b3ecc8bbacf7ac4d1cea1bbeb/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/6b1ddab23a2ac3359720cd25ea4b0de4d35830e6e74827836cec252b3f98ec42/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/c29f354dd8fd0c31c6bec6be91adf79978ee2f52c99f309b8128b53fe0b2d691/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/125c7ae42df57fce5c3fc28cfed48bc2d1a0c1e6931aa0545e7eada1987c9d9d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/a7da474e1eabdfb8fc4d80ffaba9a85d707282e744ba8782fd55b6c1285e8f8a/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/e08b9d22cf1f1ed14e3c3fb65a410b56c2dd28d377004309024789adec284a4d/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/0cd0aad47e8635cca659237e53a17ebbc953b1c554eb342925c4c43c0a9f0f1a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/ea9c97a110b0178573dea6477160d145c0db7b3f9214c8aebb851bf34530523c/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/a8e36fa48ffcb2daa3e3a9939db9d0a71c6e2ce4522037021b5e50ca62575180/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/8f6b78892e5bf6db104632be23b4406680c6cdb983e40511d715557e917d9ce3/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/4d64d228692855c7458626c45e87a49546274ede4b82a12b99a4120860801873/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/3115f2196e4becd3462ae313b6f03acacbbdef21264f8eef7df6c30239582d8f/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/f8e10099fe7a1a6f55c86c2514a8b5b7788563ebdd3ffbf2c1b724fa4e7d06fe/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/3b6a729e19fef22086819af88a4271f40a62ef085cd389178d530bf8bdd59034/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/dca138b89541b7985d20b50f87352f3e160f40cd0ee79796af761fe79f971efa/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/5c41060059f5a052e3ea89de28a7a2c636fff7f852abc7e1eeb1298f5a684412/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/a1b3f971405132a831b85b2425b694b472fb85686bbf5692e961f267c3f7a580/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/c52c18bdd10e725f0a106fd523008aa9985d03b813fa46a9e81e0d3d7c2eb67d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/6b0535045624b6bc488d1954991fe1150d66539675bed315c2f8d7ed085b0669/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/7186740b5dc22cb1398d7795d6c2e4202282f205baac80539cb652be44de280d/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/a0b5a0785e1361b354d92c92eb4a092365b01f34e79a927fa103631b883c9a88/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/9867af926b48424ac58c63e3d910519fd61bb281ae9db4a7847f94ea72449aa9/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/bb64e9e6c0eaeb9bc46f7ca711033f583880daf20c5b90976426b0fc905c7f7c/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/9c21b113e9c8e23ec838d0590268daecf65778a05eb16d255dd85b6900dff23f/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/2ecab3847a8f87b4984d9c1a67ae6fcfa46971739d3917d0795c0be59889fa97/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/97347ff342f433a1f3d18aa89408d11f73065ba8ad8c9118339c633f427d3be6/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/fa696daf557f7d9b8ac15cba3fbf0e3d0393e2f517183d33a8cf88b69152c535/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/c0c1216d787c7ef04f4b8e53205e49014fc03a06b4debaf010e0285d8fd90b32/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/d2f9ff463508a6d7b5a1c5df4f05a834e3994d75cd4a5dd4cafb8cd55421566a/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/aa003c362f859a66346d854b07f8750d43bf82c75db3fe4fb22ae7e6f8cf3ef1/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/95ec3fb2844dbbf7aaced6e4f8f59d1a9315c3479ccd6ed497b5206d520f703c/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/46d8c90b0d16073db4b1dcae20b319f1397de5e63eb6a4694a8c7f3a34abf72e/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/10b6e025708e9a23868e8675964d80a4f99b4ffa5923a267fb1b266b69f3a479/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/6ab183a50f75a0ebda21e0551e31055de078fbde2a6a9a88eca22644a6997296/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/53b038be90c93376d7fab05bb8bb0082e8594df225ca1bc99915af73319c08a0/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/1dab34f3b0adabb4ce828fbdbb70115bd15dab9d23d4a614fc5af6bb195bea1b/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/aef8de04da4d000b0c0c453b46b14b072ebfd881c9491990603a9fcc9f76b2cc/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/afdd0914e759d35f0f214c488dba357fbf7101b8dbf9d4c702dc5e508c34f62c/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/9e94dace9a69b13a78871995e16c484b1be5310c835e581b8da9a0d94be553b2/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/69693018944cfe61274e067f0f84c6a6478ea7c45b1d81525b5706ed9adea657/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/04b7f79ce3afd435158016459b052ea8ded5ecbe4c39f56ab41595dda255f117/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/f9ca818fd8aa11846d0749d4ceb4a4bf0df393b8f25932fb18bc039a2dd3facd/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/de545c70557a305a43697f420830e61ef2f1312a4c6b155ab4f87138186107ee/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/1af72fc8221110c9d45ba3253ab2f15512b197a093e726fcf52282603fe30899/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/8e8e302421d93784e7c75005729643d034fed5ce500a07af300e5aa62fd49ed3/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/8752f1cd26798660634291f3d954d925d4461e0141787e2750b3ee73398d1f31/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/1395cf8ac7d7e2dbfe5fdb7fa6bb410dba7cee0e3d2b70f4e8ef5104ca677791/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7272ecf535f5ecaf7f300e57ebec089b0c331f4b929a79db3236d3c4999a3428/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1e0009b36fcd4a72087f45ee829e4721eb03926c73a698cc0b154a6b63f0e61f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6d23375649abfd3093290d1a8ae5c81d97d99072d24642803b924954d1b409f1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/73fbbc31fecb37e7745113921f9d8b7c2403307e859e6d8e6feafd363619e025/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/866427c54edf7d9f6e7e3862d17376c3a9188ac51131e530bac0a09e7cd8d574/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/539ab847c31c66afef1cd48fbae1d70328bd5a7ff2a555b72666bd530800a57f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/35fead138e4f88b94f37c768e12c2241cc7618086e08c503287fc01c91b09347/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/93b9e5320caed1b508823300dc110c40b0a9172b3b3fb53fda353312610db7cd/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c8eb44f289f2e9e4ac59c07da56580f864d3fdb0d1936c06a66224b6c8de6800/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.945283 5946 manager.go:217] Machine: {Timestamp:2026-11-03 16:12:09.944813472 +0000 UTC m=+0.065866285 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097344 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24928d62b2703c26b8aa9347942485 SystemUUID:ec24928d-62b2-703c-26b8-aa9347942485 BootID:a46a415a-0dac-4aa4-a86b-3668d7bf5e5a Filesystems:[{Device:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0710ffa9a941cc18f494e9dd2af469e43af2875eda5f24677845d7ec0c82d9b2/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:373 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8540df11-165c-4b01-b8a3-4042a8f6fdab/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:429 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71b2223e985e4191587f83be652b00056d583876f94db1ba5ebc2b750823d2de/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:372 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05b3881e-7078-461d-9c94-69d049ccc417/volumes/kubernetes.io~projected/kube-api-access-8pxqv DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:376 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3/volumes/kubernetes.io~projected/kube-api-access-7648n DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27547d86ebff3bbf4b258bacccf495303f620bfae8ba86730882b7bde3f8e921/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~projected/kube-api-access-lhtmh DeviceMajor:0 DeviceMinor:260 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4d113c1a2471e4a44779b05c6a78480ce7c8147b02d073e989def2314140f2c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28b1f276c9a8cc016ccc7119edebc5f02f396ad024aac0fdad27a3009cf07301/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/446f06d2e92ad81b4ec59682f6b21398563e75b42d49440ea4b7342652856a0b/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7da5e2a9-8f58-4ee1-8a00-c443413a249c/volumes/kubernetes.io~projected/kube-api-access-8czjl DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9242237ffc40dc6826d7f72c316274ea50e9b61bc516b42ee1341913f7100716/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:374 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab478348-e9ee-4b58-a398-422dd7e03f0b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:425 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3940a1ec9ee29b1dbf2f584e8b94b030a938a43b492a0bf8996d34f977202cb6/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cedb2971-5861-4832-bda4-242c206a0822/volumes/kubernetes.io~projected/kube-api-access-r5h28 DeviceMajor:0 DeviceMinor:325 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4452adf36900b0b1cd18a3c51a374294c9200a6fb22f2f2f20a3c63e81c79e5a/userdata/shm DeviceMajor:0 DeviceMinor:431 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/507f7522-ec6f-4e48-b362-68e1c9dafeb8/volumes/kubernetes.io~projected/kube-api-access-jp64h DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2218179a-fa77-4676-ad59-885f522a5039/volumes/kubernetes.io~projected/kube-api-access-j62d8 DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d437c224f8ae2c855cb246829fe1bffbebe3148e6e688cd8610a4e2a17b6fc3/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:362 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~projected/kube-api-access-jn6md DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecdd9d33-7747-417b-bff0-688639640a48/volumes/kubernetes.io~projected/kube-api-access-kr6q9 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05b3881e-7078-461d-9c94-69d049ccc417/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:200 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ebd1669d012c917e5f5a5f98e924ff4bb3b45356e03383dd033b4686e415e1c/userdata/shm DeviceMajor:0 DeviceMinor:435 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0c67fab-535b-4a8e-a336-ca475185c3dd/volumes/kubernetes.io~projected/kube-api-access-ltrdq DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/289d2a22-071a-436a-afa6-8bfecabaa60c/volumes/kubernetes.io~projected/kube-api-access-c56sh DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2218179a-fa77-4676-ad59-885f522a5039/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c0428d5-c0b3-45e7-8c92-9fdcdc798786/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae082eed5008916f48815573bdc06e2bb3521ac656ac4c5269ff2348815626c6/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83749d3087fca32ebacf984bd6d694d91fac851df15604add05aff0a98ab76ca/userdata/shm DeviceMajor:0 DeviceMinor:433 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~projected/kube-api-access-rx4nn DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9189a7df-aae8-4728-be2b-569f2cffebc3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a12181362c94a76b51bb5c5bcfe89314053884f6ab2141085f7ab9036a2a26be/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:257 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~projected/kube-api-access-8xxqq DeviceMajor:0 DeviceMinor:377 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:292 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:371 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b32d83ba628570afdfca0d09fb9053490d77ac12f9f272b573b2cf393b32481c/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/caf14ab9-d061-4cfb-bc58-68ecf17a10a7/volumes/kubernetes.io~projected/kube-api-access-42gvz DeviceMajor:0 DeviceMinor:430 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebf950b28b0574653f84097e73ed1ec5c049bed29fa96063cf4289dded008f6f/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a5bf72fdce77c3fe97c071e3a35ab348b54ec0f28dc84348b84a9b752a049ce/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~projected/kube-api-access-7xrll DeviceMajor:0 DeviceMinor:294 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a7cc691cd382057664b54b18d4c13cf31888e63190e40eded9609f9e29f97c7/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~projected/kube-api-access-mvlq9 DeviceMajor:0 DeviceMinor:254 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:370 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:375 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43230f8a-cf14-4872-b51a-a4f675f947fb/volumes/kubernetes.io~projected/kube-api-access-qwd77 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe7ec78289e3c0d5c945b9ea4d3999bf62c7611201c21f4d95150d6516e442a7/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0cc589b8a988fb2e7c922b797125feaaa9fe29d96140442045106196f587069/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61daa065-4042-47db-add5-514189226920/volumes/kubernetes.io~projected/kube-api-access-skprg DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:295 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87c40c67d97f6d4f229fe3b64865f4d0bb39bf93f4f9519a8eea8f9fd8e302d8/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:368 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cedb2971-5861-4832-bda4-242c206a0822/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8540df11-165c-4b01-b8a3-4042a8f6fdab/volumes/kubernetes.io~projected/kube-api-access-6fjnh DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e29f1414-146f-4bb2-a62c-a29b6f54ff75/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:369 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-117 DeviceMajor:0 DeviceMinor:117 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3833015d-9c9e-49d7-8630-a4b118d57ab6/volumes/kubernetes.io~projected/kube-api-access-w5fdm DeviceMajor:0 DeviceMinor:255 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/234cc11d-353c-4f0c-ab3b-4616ddd0b26d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9344a7-9120-40ef-8671-5bc1fd4de206/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7da5e2a9-8f58-4ee1-8a00-c443413a249c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:123 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d28b27f98f82bd7dd24b18d0f8f41eec8a911886466d382660e839f75c09ba04/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61a83e02-ce4d-4df6-bc29-368ec906215f/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0710ffa9a941cc1 MacAddress:2a:3c:7d:5b:ae:13 Speed:10000 Mtu:8901} {Name:4452adf36900b0b MacAddress:da:76:8d:e5:a2:ae Speed:10000 Mtu:8901} {Name:71b2223e985e419 MacAddress:ae:17:5d:e3:20:66 Speed:10000 Mtu:8901} {Name:83749d3087fca32 MacAddress:de:d4:df:50:0e:ee Speed:10000 Mtu:8901} {Name:9242237ffc40dc6 MacAddress:f6:f0:50:d6:49:01 Speed:10000 Mtu:8901} {Name:9a7cc691cd38205 MacAddress:9e:b6:29:b9:74:21 Speed:10000 Mtu:8901} {Name:9d437c224f8ae2c MacAddress:ce:54:87:49:d1:65 Speed:10000 Mtu:8901} {Name:9ebd1669d012c91 MacAddress:4a:92:8a:4b:1e:4e Speed:10000 Mtu:8901} {Name:b32d83ba628570a MacAddress:8e:35:6d:56:e7:fb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ee:8e:d4:c5 Speed:0 Mtu:9001} {Name:br-int MacAddress:0e:d1:dd:5a:13:2f Speed:0 Mtu:8901} {Name:d28b27f98f82bd7 MacAddress:5a:cd:42:21:06:f1 Speed:10000 Mtu:8901} {Name:ebf950b28b05746 MacAddress:42:e4:a1:cb:4c:28 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ee:8e:d4:c5 Speed:0 Mtu:9001} {Name:f0cc589b8a988fb MacAddress:2e:36:f6:2a:ea:9f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4e:8e:0f:5b:de:15 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:b9:56:9b:84:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.945697 5946 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.945745 5946 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946037 5946 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946061 5946 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-216.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946185 5946 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946193 5946 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946222 5946 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946244 5946 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946314 5946 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946468 5946 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946580 5946 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946614 5946 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946627 5946 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946636 5946 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946645 5946 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.946653 5946 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.947430 5946 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:12:09.947631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.947443 5946 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:12:09.948641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.947941 5946 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:12:09.950560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950545 5946 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:12:09.950560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950559 5946 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:12:09.950658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950649 5946 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 16:12:09.950842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950831 5946 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:12:09.950875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950861 5946 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:12:09.950967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950956 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950970 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950977 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950982 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950988 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950994 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.950999 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.951005 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.951011 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.951017 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:12:09.951025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.951032 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:12:09.951535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.951042 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:12:09.951535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.951067 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:12:09.951535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.951073 5946 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:12:09.954507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.954492 5946 server.go:1263] "Started kubelet" Nov 03 16:12:09.954601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.954557 5946 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:12:09.954650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.954580 5946 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:12:09.955020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.954649 5946 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:12:09.955035 ip-10-0-135-216 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:12:09.955767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.955323 5946 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:12:09.959718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.959697 5946 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.961269 5946 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.961471 5946 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.961632 5946 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:04:55 +0000 UTC" deadline="2028-08-17 15:05:20.623177726 +0000 UTC" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.961650 5946 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15670h53m10.661530917s" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.961812 5946 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.961823 5946 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.961894 5946 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:09.962420 5946 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.962436 5946 factory.go:55] Registering systemd factory Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.962452 5946 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964383 5946 factory.go:153] Registering CRI-O factory Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964400 5946 factory.go:223] Registration of the crio container factory successfully Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964431 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" volumeName="kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964466 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05b3881e-7078-461d-9c94-69d049ccc417" volumeName="kubernetes.io/secret/05b3881e-7078-461d-9c94-69d049ccc417-cert" seLinuxMountContext="" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964480 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedb2971-5861-4832-bda4-242c206a0822" volumeName="kubernetes.io/projected/cedb2971-5861-4832-bda4-242c206a0822-kube-api-access-r5h28" seLinuxMountContext="" Nov 03 16:12:09.964484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964492 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964504 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964518 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-db" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964455 5946 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964530 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" volumeName="kubernetes.io/projected/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-kube-api-access-lhtmh" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964598 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3833015d-9c9e-49d7-8630-a4b118d57ab6" volumeName="kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964625 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8540df11-165c-4b01-b8a3-4042a8f6fdab" volumeName="kubernetes.io/projected/8540df11-165c-4b01-b8a3-4042a8f6fdab-kube-api-access-6fjnh" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964637 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964649 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43230f8a-cf14-4872-b51a-a4f675f947fb" volumeName="kubernetes.io/projected/43230f8a-cf14-4872-b51a-a4f675f947fb-kube-api-access-qwd77" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964662 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a83e02-ce4d-4df6-bc29-368ec906215f" volumeName="kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964674 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da5e2a9-8f58-4ee1-8a00-c443413a249c" volumeName="kubernetes.io/projected/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-api-access-8czjl" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964686 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9189a7df-aae8-4728-be2b-569f2cffebc3" volumeName="kubernetes.io/configmap/9189a7df-aae8-4728-be2b-569f2cffebc3-konnectivity-ca" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964702 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" volumeName="kubernetes.io/empty-dir/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-textfile" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964714 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="507f7522-ec6f-4e48-b362-68e1c9dafeb8" volumeName="kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964726 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964739 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a83e02-ce4d-4df6-bc29-368ec906215f" volumeName="kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964750 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config-out" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964761 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964772 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61daa065-4042-47db-add5-514189226920" volumeName="kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-cni-binary-copy" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964784 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da5e2a9-8f58-4ee1-8a00-c443413a249c" volumeName="kubernetes.io/configmap/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964796 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3833015d-9c9e-49d7-8630-a4b118d57ab6" volumeName="kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964808 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0428d5-c0b3-45e7-8c92-9fdcdc798786" volumeName="kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-tuned" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964833 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="507f7522-ec6f-4e48-b362-68e1c9dafeb8" volumeName="kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964846 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964858 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964870 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecdd9d33-7747-417b-bff0-688639640a48" volumeName="kubernetes.io/projected/ecdd9d33-7747-417b-bff0-688639640a48-kube-api-access-kr6q9" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964896 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2218179a-fa77-4676-ad59-885f522a5039" volumeName="kubernetes.io/secret/2218179a-fa77-4676-ad59-885f522a5039-metrics-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964912 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" volumeName="kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964924 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43230f8a-cf14-4872-b51a-a4f675f947fb" volumeName="kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-config" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964937 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43230f8a-cf14-4872-b51a-a4f675f947fb" volumeName="kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-script-lib" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964949 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/empty-dir/5e9344a7-9120-40ef-8671-5bc1fd4de206-tmp" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964962 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61daa065-4042-47db-add5-514189226920" volumeName="kubernetes.io/projected/61daa065-4042-47db-add5-514189226920-kube-api-access-skprg" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964974 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da5e2a9-8f58-4ee1-8a00-c443413a249c" volumeName="kubernetes.io/secret/7da5e2a9-8f58-4ee1-8a00-c443413a249c-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964988 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964999 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8ec6fc-a61e-4d85-b9c8-f594647b0a96" volumeName="kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965011 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8ec6fc-a61e-4d85-b9c8-f594647b0a96" volumeName="kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965023 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8ec6fc-a61e-4d85-b9c8-f594647b0a96" volumeName="kubernetes.io/empty-dir/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-audit-log" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965035 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61daa065-4042-47db-add5-514189226920" volumeName="kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-multus-daemon-config" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965046 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="289d2a22-071a-436a-afa6-8bfecabaa60c" volumeName="kubernetes.io/projected/289d2a22-071a-436a-afa6-8bfecabaa60c-kube-api-access-c56sh" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965059 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da5e2a9-8f58-4ee1-8a00-c443413a249c" volumeName="kubernetes.io/empty-dir/7da5e2a9-8f58-4ee1-8a00-c443413a249c-data-volume" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965071 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965083 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-tls-assets" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965094 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-grpc-tls" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965104 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecdd9d33-7747-417b-bff0-688639640a48" volumeName="kubernetes.io/empty-dir/ecdd9d33-7747-417b-bff0-688639640a48-tmp-dir" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965116 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" volumeName="kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965128 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43230f8a-cf14-4872-b51a-a4f675f947fb" volumeName="kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-env-overrides" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965143 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43230f8a-cf14-4872-b51a-a4f675f947fb" volumeName="kubernetes.io/secret/43230f8a-cf14-4872-b51a-a4f675f947fb-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965155 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="507f7522-ec6f-4e48-b362-68e1c9dafeb8" volumeName="kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-binary-copy" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965166 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a83e02-ce4d-4df6-bc29-368ec906215f" volumeName="kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965178 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedb2971-5861-4832-bda4-242c206a0822" volumeName="kubernetes.io/secret/cedb2971-5861-4832-bda4-242c206a0822-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965189 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05b3881e-7078-461d-9c94-69d049ccc417" volumeName="kubernetes.io/projected/05b3881e-7078-461d-9c94-69d049ccc417-kube-api-access-8pxqv" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965200 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3833015d-9c9e-49d7-8630-a4b118d57ab6" volumeName="kubernetes.io/configmap/3833015d-9c9e-49d7-8630-a4b118d57ab6-metrics-client-ca" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965230 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3833015d-9c9e-49d7-8630-a4b118d57ab6" volumeName="kubernetes.io/projected/3833015d-9c9e-49d7-8630-a4b118d57ab6-kube-api-access-w5fdm" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965241 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8ec6fc-a61e-4d85-b9c8-f594647b0a96" volumeName="kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-client-ca-bundle" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965261 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/configmap/5e9344a7-9120-40ef-8671-5bc1fd4de206-metrics-client-ca" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965274 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a83e02-ce4d-4df6-bc29-368ec906215f" volumeName="kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-metrics-client-ca" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965286 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.964560 5946 factory.go:103] Registering Raw factory Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965298 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2218179a-fa77-4676-ad59-885f522a5039" volumeName="kubernetes.io/projected/2218179a-fa77-4676-ad59-885f522a5039-kube-api-access-j62d8" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965309 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0428d5-c0b3-45e7-8c92-9fdcdc798786" volumeName="kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-tmp" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965320 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/projected/5e9344a7-9120-40ef-8671-5bc1fd4de206-kube-api-access-7xrll" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965335 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965349 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a83e02-ce4d-4df6-bc29-368ec906215f" volumeName="kubernetes.io/empty-dir/61a83e02-ce4d-4df6-bc29-368ec906215f-volume-directive-shadow" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965360 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9189a7df-aae8-4728-be2b-569f2cffebc3" volumeName="kubernetes.io/secret/9189a7df-aae8-4728-be2b-569f2cffebc3-agent-certs" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965373 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0c67fab-535b-4a8e-a336-ca475185c3dd" volumeName="kubernetes.io/configmap/c0c67fab-535b-4a8e-a336-ca475185c3dd-serviceca" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965384 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 16:12:09.965352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965397 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2218179a-fa77-4676-ad59-885f522a5039" volumeName="kubernetes.io/configmap/2218179a-fa77-4676-ad59-885f522a5039-config-volume" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965408 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2218179a-fa77-4676-ad59-885f522a5039" volumeName="kubernetes.io/empty-dir/2218179a-fa77-4676-ad59-885f522a5039-tmp-dir" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965420 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" volumeName="kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-metrics-client-ca" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965324 5946 manager.go:1199] Started watching for new ooms in manager Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965432 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0428d5-c0b3-45e7-8c92-9fdcdc798786" volumeName="kubernetes.io/projected/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-kube-api-access-jn6md" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965445 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8ec6fc-a61e-4d85-b9c8-f594647b0a96" volumeName="kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965456 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965468 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a83e02-ce4d-4df6-bc29-368ec906215f" volumeName="kubernetes.io/projected/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-api-access-mvlq9" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965480 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-kube-api-access-8xxqq" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965491 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="289d2a22-071a-436a-afa6-8bfecabaa60c" volumeName="kubernetes.io/configmap/289d2a22-071a-436a-afa6-8bfecabaa60c-iptables-alerter-script" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965502 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965523 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965534 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965546 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="507f7522-ec6f-4e48-b362-68e1c9dafeb8" volumeName="kubernetes.io/projected/507f7522-ec6f-4e48-b362-68e1c9dafeb8-kube-api-access-jp64h" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965557 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8ec6fc-a61e-4d85-b9c8-f594647b0a96" volumeName="kubernetes.io/projected/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-kube-api-access-rx4nn" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965568 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8ec6fc-a61e-4d85-b9c8-f594647b0a96" volumeName="kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965580 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-grpc-tls" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965591 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab478348-e9ee-4b58-a398-422dd7e03f0b" volumeName="kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965602 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acc9fe0f-08e8-4cb1-bab7-ace0f47412c3" volumeName="kubernetes.io/projected/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kube-api-access-7648n" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965612 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0c67fab-535b-4a8e-a336-ca475185c3dd" volumeName="kubernetes.io/projected/c0c67fab-535b-4a8e-a336-ca475185c3dd-kube-api-access-ltrdq" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965623 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caf14ab9-d061-4cfb-bc58-68ecf17a10a7" volumeName="kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965634 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9344a7-9120-40ef-8671-5bc1fd4de206" volumeName="kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965645 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8540df11-165c-4b01-b8a3-4042a8f6fdab" volumeName="kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965657 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-web-config" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965677 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965689 5946 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29f1414-146f-4bb2-a62c-a29b6f54ff75" volumeName="kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965700 5946 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965707 5946 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:12:09.967731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.965796 5946 manager.go:319] Starting recovery of all containers Nov 03 16:12:09.994768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.994713 5946 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:12:09.996381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.996358 5946 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:12:09.996381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.996380 5946 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:12:09.997871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.996399 5946 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:12:09.997871 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:09.996442 5946 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 16:12:09.999076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.999054 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:12:09.999237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:09.999194 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:12:10.006626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.006598 5946 generic.go:356] "Generic (PLEG): container finished" podID="234cc11d-353c-4f0c-ab3b-4616ddd0b26d" containerID="5dacbcee14cc7f96b546deaa53217704e453e0f009dce2225b8a3db0dbb6cdd1" exitCode=0 Nov 03 16:12:10.016776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.016750 5946 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="5fbab535fff0a5bb694f778bc6b6b368f9a4b415b3930955ff263d4588aca5ad" exitCode=0 Nov 03 16:12:10.016776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.016774 5946 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="4e250fcfcb46d38445903394a2890223e5ab0f42fc1d9c0c4f55ed7f61100085" exitCode=0 Nov 03 16:12:10.016908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.016785 5946 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="d11c96055334defe42aabdc8299a61d172b3635c7fe1fba13fd57eb1af6373c3" exitCode=0 Nov 03 16:12:10.016908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.016792 5946 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="0f63bd236223623fa651c982cfed1bd807ad0d3085ecc010ea6d65dfe9bd490d" exitCode=0 Nov 03 16:12:10.016908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.016800 5946 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="7ec88c02f1303eb549c2608ba3bebcf0226a11bf455126268c72d969dbc7ca58" exitCode=0 Nov 03 16:12:10.016908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.016807 5946 generic.go:356] "Generic (PLEG): container finished" podID="507f7522-ec6f-4e48-b362-68e1c9dafeb8" containerID="2160ead286f403d511ffe0a82f4587943e13a788a871fee07cfd34dc29a689b9" exitCode=0 Nov 03 16:12:10.027920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.027896 5946 generic.go:356] "Generic (PLEG): container finished" podID="e29f1414-146f-4bb2-a62c-a29b6f54ff75" containerID="7ba7cfe37605f154077c556bc745f887059ae5bf0d3e847100123354654f7e0f" exitCode=0 Nov 03 16:12:10.035919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.035900 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:12:10.035994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.035930 5946 generic.go:356] "Generic (PLEG): container finished" podID="61daa065-4042-47db-add5-514189226920" containerID="5126f39337fe32c995b9a7d0d6f39930b33c77351e08ea79a1b33e0388ea31ea" exitCode=1 Nov 03 16:12:10.039104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.039074 5946 generic.go:356] "Generic (PLEG): container finished" podID="fed4b634a3c30570b86550ce73c63486" containerID="6a4d9a0565009bd7244eb01bca4f02fde4d715b8f49b05adc1b98ca11194c5ad" exitCode=0 Nov 03 16:12:10.062530 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:10.062463 5946 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:12:10.074022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.074005 5946 manager.go:324] Recovery completed Nov 03 16:12:10.095125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.095100 5946 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:12:10.097292 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:10.097274 5946 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 16:12:10.098054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.098035 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:12:10.098166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.098071 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:12:10.098166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.098085 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:12:10.100735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100714 5946 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:12:10.100735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100734 5946 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:12:10.100886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100754 5946 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:12:10.100886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100726 5946 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:12:10.101004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100917 5946 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 16:12:10.101004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100933 5946 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 16:12:10.101004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100956 5946 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 16:12:10.101004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100964 5946 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 16:12:10.101004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100973 5946 policy_none.go:50] "Start" Nov 03 16:12:10.101004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100986 5946 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:12:10.101004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.100998 5946 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:12:10.101339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.101176 5946 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 16:12:10.101339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.101187 5946 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 16:12:10.101339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.101195 5946 policy_none.go:44] "Start" Nov 03 16:12:10.101766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.101747 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:12:10.101821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.101778 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:12:10.101821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.101789 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:12:10.105923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.105898 5946 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:12:10.109926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.109910 5946 manager.go:341] "Starting Device Plugin manager" Nov 03 16:12:10.110039 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:10.109936 5946 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:12:10.110039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.109946 5946 server.go:87] "Starting device plugin registration server" Nov 03 16:12:10.110187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.110175 5946 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:12:10.110265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.110188 5946 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:12:10.110319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.110273 5946 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:12:10.110397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.110360 5946 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:12:10.110494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.110404 5946 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:12:10.110494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.110411 5946 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:12:10.110591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.110459 5946 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:12:10.110394218Z","UUID":"223d2f45-cdc0-44d9-9423-4492a83d74fc","Handler":null,"Name":"","Endpoint":""} Nov 03 16:12:10.111794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.111773 5946 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:12:10.111871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.111813 5946 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:12:10.112046 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:10.112021 5946 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:12:10.112140 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:10.112067 5946 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-216.us-west-2.compute.internal\" not found" Nov 03 16:12:10.165109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.165079 5946 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:12:10.165236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.165188 5946 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 16:12:10.166626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.166612 5946 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:12:10.210649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.210624 5946 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:12:10.211790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.211773 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:12:10.211864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.211802 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:12:10.211864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.211812 5946 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-216.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:12:10.211864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.211857 5946 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.215074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.215058 5946 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:12:10.266432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.266410 5946 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:12:10.298344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.298304 5946 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal"] Nov 03 16:12:10.298987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.298969 5946 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.299078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.299008 5946 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.299078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.299035 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"fed4b634a3c30570b86550ce73c63486","Type":"ContainerStarted","Data":"e6e0f76ea250eb1acc1c8b9a64c61e373c124bc926aed1cf77e78f2e47d7b2d4"} Nov 03 16:12:10.299167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.299090 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"fed4b634a3c30570b86550ce73c63486","Type":"ContainerDied","Data":"6a4d9a0565009bd7244eb01bca4f02fde4d715b8f49b05adc1b98ca11194c5ad"} Nov 03 16:12:10.299167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.299104 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"fed4b634a3c30570b86550ce73c63486","Type":"ContainerStarted","Data":"87c40c67d97f6d4f229fe3b64865f4d0bb39bf93f4f9519a8eea8f9fd8e302d8"} Nov 03 16:12:10.299167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.299130 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"4357665f1982cbd14d16f18df231ee45","Type":"ContainerStarted","Data":"00d7d75730dfe4f13e0cc9166291e3f3939cc2d7c1e17f86e0da0daea084e5d2"} Nov 03 16:12:10.299167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.299142 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" event={"ID":"4357665f1982cbd14d16f18df231ee45","Type":"ContainerStarted","Data":"b19168ee01da7cd2e38ac173f1f3ee6c451e5f1afe53cf1a2d53a94de96c8114"} Nov 03 16:12:10.325995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.325944 5946 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.326053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.326000 5946 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.361406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.361380 5946 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:12:10.361493 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:10.361431 5946 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.365154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.365138 5946 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:12:10.365242 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:10.365179 5946 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.366266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.366244 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.366316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.366274 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.366316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.366293 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4357665f1982cbd14d16f18df231ee45-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"4357665f1982cbd14d16f18df231ee45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.466972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.466945 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.466972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.466977 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4357665f1982cbd14d16f18df231ee45-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"4357665f1982cbd14d16f18df231ee45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.467180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.466993 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.467180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.467030 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.467180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.467063 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed4b634a3c30570b86550ce73c63486-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"fed4b634a3c30570b86550ce73c63486\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.467180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.467066 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4357665f1982cbd14d16f18df231ee45-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" (UID: \"4357665f1982cbd14d16f18df231ee45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:10.946903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:10.946871 5946 apiserver.go:52] "Watching apiserver" Nov 03 16:12:11.054732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.054703 5946 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:11.054732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.054722 5946 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:12:11.054930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.054829 5946 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:11.055327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.055307 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw","openshift-ingress-canary/ingress-canary-wc6dg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal","openshift-multus/multus-8r945","kube-system/konnectivity-agent-hm72k","openshift-cluster-node-tuning-operator/tuned-54n6m","openshift-insights/insights-runtime-extractor-qbst4","openshift-monitoring/node-exporter-hfllm","openshift-monitoring/metrics-server-d7dfbc995-gq2ct","openshift-monitoring/thanos-querier-5b684bf768-ghn5f","openshift-monitoring/monitoring-plugin-77f4f76594-nw79m","openshift-network-diagnostics/network-check-target-67tfw","openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql","openshift-multus/multus-additional-cni-plugins-kv69g","openshift-image-registry/node-ca-bcgxn","openshift-monitoring/prometheus-k8s-0","openshift-ovn-kubernetes/ovnkube-node-gsdp4","kube-system/global-pull-secret-syncer-5dbjc","openshift-multus/network-metrics-daemon-fddzx","openshift-network-operator/iptables-alerter-x86gk","kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd","openshift-dns/dns-default-xg99b","openshift-dns/node-resolver-xt54q"] Nov 03 16:12:11.070899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.070874 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6q9\" (UniqueName: \"kubernetes.io/projected/ecdd9d33-7747-417b-bff0-688639640a48-kube-api-access-kr6q9\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:12:11.071052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.070903 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:12:11.071052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.070920 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.071052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.070937 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-sys\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.071219 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071183 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.071261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071229 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-ovn\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.071292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071269 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.071327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071286 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0c67fab-535b-4a8e-a336-ca475185c3dd-host\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:12:11.071327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071314 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecdd9d33-7747-417b-bff0-688639640a48-tmp-dir\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:12:11.071382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071346 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.071411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071389 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-conf-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.071451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071438 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecdd9d33-7747-417b-bff0-688639640a48-tmp-dir\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:12:11.071505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071493 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-lib-modules\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.071536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071523 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-config\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.071573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071556 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:12:11.071604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071583 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-system-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.071632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071623 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-socket-dir-parent\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.071665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071638 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-hostroot\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.071665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071652 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-systemd\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.071721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071666 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.071721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071681 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-kubelet\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.071721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071719 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0c67fab-535b-4a8e-a336-ca475185c3dd-serviceca\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:12:11.071803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071745 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-system-cni-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.071803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071785 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9189a7df-aae8-4728-be2b-569f2cffebc3-agent-certs\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:12:11.071862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071804 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7648n\" (UniqueName: \"kubernetes.io/projected/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kube-api-access-7648n\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.071862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071844 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-systemd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.071920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071879 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp64h\" (UniqueName: \"kubernetes.io/projected/507f7522-ec6f-4e48-b362-68e1c9dafeb8-kube-api-access-jp64h\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.071920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071908 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-kubelet\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.071975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071924 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-systemd-units\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.071975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071948 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-netns\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071980 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-node-log\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.071999 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrdq\" (UniqueName: \"kubernetes.io/projected/c0c67fab-535b-4a8e-a336-ca475185c3dd-kube-api-access-ltrdq\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:12:11.072083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072033 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-sys-fs\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.072114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072097 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-conf\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072134 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.072192 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072179 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-modprobe-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072245 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-host\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072279 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecdd9d33-7747-417b-bff0-688639640a48-hosts-file\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:12:11.072324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072294 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-netns\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.072324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072310 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6md\" (UniqueName: \"kubernetes.io/projected/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-kube-api-access-jn6md\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072326 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-kubelet-config\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.072379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072341 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-log-socket\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072371 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-netd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072398 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072452 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skprg\" (UniqueName: \"kubernetes.io/projected/61daa065-4042-47db-add5-514189226920-kube-api-access-skprg\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.072517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072499 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-multus-certs\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.072605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072593 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd77\" (UniqueName: \"kubernetes.io/projected/43230f8a-cf14-4872-b51a-a4f675f947fb-kube-api-access-qwd77\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072710 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-kubernetes\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072749 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-run\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072773 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.072818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072790 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-bin\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072823 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-tuned\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072865 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-dbus\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.072910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072892 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-etc-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072905 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-tuned\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.072941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072918 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-env-overrides\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.072941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072938 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/289d2a22-071a-436a-afa6-8bfecabaa60c-iptables-alerter-script\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:12:11.073021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072976 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/289d2a22-071a-436a-afa6-8bfecabaa60c-host-slash\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:12:11.073021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.072999 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-multus-daemon-config\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073030 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-etc-kubernetes\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073062 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.073132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073083 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.073132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073100 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-script-lib\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.073132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073122 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjnh\" (UniqueName: \"kubernetes.io/projected/8540df11-165c-4b01-b8a3-4042a8f6fdab-kube-api-access-6fjnh\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:12:11.073231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073169 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-socket-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.073231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073194 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-device-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.073294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073249 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9189a7df-aae8-4728-be2b-569f2cffebc3-konnectivity-ca\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:12:11.073294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073274 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.073350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073293 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073365 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-cni-binary-copy\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073386 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-k8s-cni-cncf-io\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073422 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-bin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073455 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-registration-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.073498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073482 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-slash\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.073540 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073502 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-var-lib-kubelet\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.073572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073537 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-cnibin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073563 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-multus\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.073631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073581 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-tmp\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.073631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073616 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-var-lib-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.073687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073642 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-tmp\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.073687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073671 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cnibin\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.073741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073686 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysconfig\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.073741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073719 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43230f8a-cf14-4872-b51a-a4f675f947fb-ovn-node-metrics-cert\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.073796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.073753 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56sh\" (UniqueName: \"kubernetes.io/projected/289d2a22-071a-436a-afa6-8bfecabaa60c-kube-api-access-c56sh\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:12:11.111273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111247 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:12:11.111273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111247 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.111460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111264 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:12:11.111545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111528 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.111605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111597 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:12:11.111653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111595 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:12:11.111653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111624 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tk7qx\"" Nov 03 16:12:11.111653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111650 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x72k5\"" Nov 03 16:12:11.111794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111749 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.111794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111758 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-config\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.111794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111785 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:12:11.111927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111843 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:12:11.111927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111894 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxk2h\"" Nov 03 16:12:11.111927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111914 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.111985 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112023 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112035 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112028 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112042 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4vrm9\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112075 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112084 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9gv4t\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112076 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112075 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.112127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112120 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xzbs4\"" Nov 03 16:12:11.112457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112163 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.112457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.112292 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:12:11.113449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.113431 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-env-overrides\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.113558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.113544 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.113558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.113545 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9189a7df-aae8-4728-be2b-569f2cffebc3-konnectivity-ca\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:12:11.113626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.113586 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43230f8a-cf14-4872-b51a-a4f675f947fb-ovnkube-script-lib\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.113626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.113603 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-cni-binary-copy\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.113685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.113633 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61daa065-4042-47db-add5-514189226920-multus-daemon-config\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.114352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.114331 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43230f8a-cf14-4872-b51a-a4f675f947fb-ovn-node-metrics-cert\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.114493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.114479 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.117843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.117826 5946 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:12:11.117900 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:11.117870 5946 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:11.117900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.117884 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xhvl5\"" Nov 03 16:12:11.117957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.117830 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.117986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.117834 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:12:11.119145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.119134 5946 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:12:11.119179 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:11.119161 5946 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.us-west-2.compute.internal" Nov 03 16:12:11.122412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.122395 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.122454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.122424 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9189a7df-aae8-4728-be2b-569f2cffebc3-agent-certs\") pod \"konnectivity-agent-hm72k\" (UID: \"9189a7df-aae8-4728-be2b-569f2cffebc3\") " pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:12:11.122454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.122395 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/507f7522-ec6f-4e48-b362-68e1c9dafeb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.123540 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.123525 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/289d2a22-071a-436a-afa6-8bfecabaa60c-iptables-alerter-script\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:12:11.150509 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150488 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.150708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150690 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.150708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150703 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.150813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150716 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7zlt\"" Nov 03 16:12:11.150868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150768 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kkrgk\"" Nov 03 16:12:11.150868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150773 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.150966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150778 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:12:11.150966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150952 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.150966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150957 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:12:11.151093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150978 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zsc2l\"" Nov 03 16:12:11.151093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.150964 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4xv66\"" Nov 03 16:12:11.151181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.151169 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:12:11.151241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.151195 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rqr4v\"" Nov 03 16:12:11.152471 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.152457 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8540df11-165c-4b01-b8a3-4042a8f6fdab-metrics-certs\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:12:11.152514 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.152492 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0c67fab-535b-4a8e-a336-ca475185c3dd-serviceca\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:12:11.152567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.152554 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab478348-e9ee-4b58-a398-422dd7e03f0b-original-pull-secret\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.158333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.158321 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.158654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.158639 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:12:11.159321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159304 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.159401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159304 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:12:11.159401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159310 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.159501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159458 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:12:11.159551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159499 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4pfhl\"" Nov 03 16:12:11.159551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159535 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.159691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159668 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 16:12:11.159835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159714 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.159835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159772 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 16:12:11.159835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159832 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zz2z\"" Nov 03 16:12:11.159938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159892 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 16:12:11.160007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.159992 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 16:12:11.160051 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160042 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:12:11.160098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160075 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 16:12:11.160151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160075 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fkfj5\"" Nov 03 16:12:11.160151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160114 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 16:12:11.160151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160115 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 16:12:11.160151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160126 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x292s\"" Nov 03 16:12:11.160298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160224 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:12:11.160298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160277 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:12:11.160372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160304 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 16:12:11.160372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160315 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 16:12:11.160372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160322 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 16:12:11.160372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160341 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:12:11.160486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160382 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-94sd2\"" Nov 03 16:12:11.160787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160774 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eql3qhcmfmus4\"" Nov 03 16:12:11.160828 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160819 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 16:12:11.160866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160828 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 16:12:11.160866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160854 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t4rpz\"" Nov 03 16:12:11.160949 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.160892 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 16:12:11.161307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.161292 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:12:11.161370 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.161318 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:12:11.162665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.162650 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 16:12:11.162742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.162692 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m7glm\"" Nov 03 16:12:11.162816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.162801 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 16:12:11.162816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.162811 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2xgnz\"" Nov 03 16:12:11.163143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.163125 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 16:12:11.163190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.163150 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 16:12:11.163648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.163633 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:12:11.164396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.164374 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:12:11.164543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.164523 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bqa94k5oj8mrd\"" Nov 03 16:12:11.164676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.164655 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gswn6\"" Nov 03 16:12:11.164676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.164672 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 16:12:11.164822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.164688 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 16:12:11.164871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.164831 5946 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:12:11.168534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.168516 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 16:12:11.174059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174029 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-netd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.174146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174072 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.174146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174099 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e9344a7-9120-40ef-8671-5bc1fd4de206-metrics-client-ca\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.174289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174140 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-netd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.174289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174143 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.174289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174160 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.174289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174216 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7da5e2a9-8f58-4ee1-8a00-c443413a249c-data-volume\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.174289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174236 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-multus-certs\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.174289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174255 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-tls\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.174289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174280 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174304 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-run\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174306 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7da5e2a9-8f58-4ee1-8a00-c443413a249c-data-volume\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174302 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-multus-certs\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174336 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e9344a7-9120-40ef-8671-5bc1fd4de206-metrics-client-ca\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174354 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-run\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174369 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4nn\" (UniqueName: \"kubernetes.io/projected/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-kube-api-access-rx4nn\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174392 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174410 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czjl\" (UniqueName: \"kubernetes.io/projected/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-api-access-8czjl\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174425 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174444 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/289d2a22-071a-436a-afa6-8bfecabaa60c-host-slash\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174458 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cedb2971-5861-4832-bda4-242c206a0822-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174472 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxqv\" (UniqueName: \"kubernetes.io/projected/05b3881e-7078-461d-9c94-69d049ccc417-kube-api-access-8pxqv\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174486 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174503 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174520 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174544 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config-out\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174549 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/289d2a22-071a-436a-afa6-8bfecabaa60c-host-slash\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:12:11.174585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174582 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-ovn\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174608 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174609 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174621 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174633 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174648 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config-out\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174659 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-ovn\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174668 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174695 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-k8s-cni-cncf-io\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174719 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-bin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174721 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174744 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174765 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-k8s-cni-cncf-io\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174772 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-registration-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174786 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174794 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-bin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174816 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174823 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174797 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-slash\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174834 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174860 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-multus\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174834 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-slash\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174879 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-client-certs\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174899 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-sys\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174856 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-registration-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174919 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-cni-multus\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174922 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174971 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.174991 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-web-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175019 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0c67fab-535b-4a8e-a336-ca475185c3dd-host\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175036 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2218179a-fa77-4676-ad59-885f522a5039-tmp-dir\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175051 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175067 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-conf-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175078 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0c67fab-535b-4a8e-a336-ca475185c3dd-host\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175091 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-wtmp\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175100 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-client-certs\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175108 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175126 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-conf-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175123 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2218179a-fa77-4676-ad59-885f522a5039-tmp-dir\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175149 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7da5e2a9-8f58-4ee1-8a00-c443413a249c-crio-socket\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175170 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-web-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175176 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-socket-dir-parent\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175197 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-systemd\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175247 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175257 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175265 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-systemd\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175273 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175276 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175302 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-multus-socket-dir-parent\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175328 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175345 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175376 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175408 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175479 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-systemd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175496 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-textfile\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175514 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-client-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175540 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-systemd-units\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175554 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-run-systemd\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175558 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175564 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-netns\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175591 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175596 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-textfile\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175605 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-systemd-units\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175620 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-run-netns\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175634 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175665 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-conf\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175637 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175688 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2218179a-fa77-4676-ad59-885f522a5039-metrics-tls\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175713 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175738 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175740 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-client-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175761 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-modprobe-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175768 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175791 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecdd9d33-7747-417b-bff0-688639640a48-hosts-file\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175793 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175816 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175820 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-conf\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175817 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175834 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175857 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175871 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecdd9d33-7747-417b-bff0-688639640a48-hosts-file\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175874 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-config\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175882 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvlq9\" (UniqueName: \"kubernetes.io/projected/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-api-access-mvlq9\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175890 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2218179a-fa77-4676-ad59-885f522a5039-metrics-tls\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175908 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-modprobe-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175914 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175945 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7da5e2a9-8f58-4ee1-8a00-c443413a249c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175969 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-netns\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.175994 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-kubelet-config\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176015 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176035 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-run-netns\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176047 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-kubelet-config\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176075 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-log-socket\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176102 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-root\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176109 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-log-socket\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176117 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7da5e2a9-8f58-4ee1-8a00-c443413a249c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176135 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-grpc-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176162 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176187 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176233 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176258 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-kubernetes\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176292 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176340 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176351 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-kubernetes\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176356 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176358 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-grpc-tls\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176388 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-bin\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176407 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h28\" (UniqueName: \"kubernetes.io/projected/cedb2971-5861-4832-bda4-242c206a0822-kube-api-access-r5h28\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176425 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3833015d-9c9e-49d7-8630-a4b118d57ab6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176450 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-cni-bin\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176454 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176451 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176487 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-dbus\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176504 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-etc-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176520 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2218179a-fa77-4676-ad59-885f522a5039-config-volume\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176557 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-etc-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176565 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab478348-e9ee-4b58-a398-422dd7e03f0b-dbus\") pod \"global-pull-secret-syncer-5dbjc\" (UID: \"ab478348-e9ee-4b58-a398-422dd7e03f0b\") " pod="kube-system/global-pull-secret-syncer-5dbjc" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176584 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62d8\" (UniqueName: \"kubernetes.io/projected/2218179a-fa77-4676-ad59-885f522a5039-kube-api-access-j62d8\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176610 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-etc-kubernetes\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176615 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176628 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176615 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3833015d-9c9e-49d7-8630-a4b118d57ab6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176635 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176666 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-tls\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176670 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-etc-kubernetes\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176682 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2218179a-fa77-4676-ad59-885f522a5039-config-volume\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176686 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtmh\" (UniqueName: \"kubernetes.io/projected/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-kube-api-access-lhtmh\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176780 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysctl-d\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176788 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrll\" (UniqueName: \"kubernetes.io/projected/5e9344a7-9120-40ef-8671-5bc1fd4de206-kube-api-access-7xrll\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176830 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b3881e-7078-461d-9c94-69d049ccc417-cert\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176849 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61a83e02-ce4d-4df6-bc29-368ec906215f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176885 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-tls\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176922 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61a83e02-ce4d-4df6-bc29-368ec906215f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176941 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-socket-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176963 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-device-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.176988 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-var-lib-kubelet\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177003 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-socket-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177012 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-var-lib-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177035 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-cnibin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177046 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-var-lib-kubelet\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177051 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177054 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-var-lib-openvswitch\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177034 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-device-dir\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177082 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177105 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-cnibin\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177113 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-metrics-server-audit-profiles\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177085 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b3881e-7078-461d-9c94-69d049ccc417-cert\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177140 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxqq\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-kube-api-access-8xxqq\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177161 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cnibin\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177176 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysconfig\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177190 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-audit-log\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177193 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177231 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-sys\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177252 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-cnibin\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177254 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e9344a7-9120-40ef-8671-5bc1fd4de206-tmp\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177270 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-etc-sysconfig\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177277 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-audit-log\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177271 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177291 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-lib-modules\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177309 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e9344a7-9120-40ef-8671-5bc1fd4de206-tmp\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177317 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177322 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-sys\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177344 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-system-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177364 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-hostroot\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177369 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-metrics-server-audit-profiles\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177373 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-lib-modules\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177379 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-kubelet\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177405 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-system-cni-dir\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177409 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-host-kubelet\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.177351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177415 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177437 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-hostroot\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177446 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-system-cni-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177477 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/507f7522-ec6f-4e48-b362-68e1c9dafeb8-system-cni-dir\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177478 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177499 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177522 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-kubelet\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177578 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-node-log\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177596 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e9344a7-9120-40ef-8671-5bc1fd4de206-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177614 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61daa065-4042-47db-add5-514189226920-host-var-lib-kubelet\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177616 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-metrics-client-ca\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177634 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177638 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43230f8a-cf14-4872-b51a-a4f675f947fb-node-log\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177644 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fdm\" (UniqueName: \"kubernetes.io/projected/3833015d-9c9e-49d7-8630-a4b118d57ab6-kube-api-access-w5fdm\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177661 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-sys-fs\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177675 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-host\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177689 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177704 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177768 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-metrics-client-ca\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177773 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-sys-fs\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177830 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-host\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.180699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.177887 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3833015d-9c9e-49d7-8630-a4b118d57ab6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.201795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.201749 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 16:12:11.202130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202117 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7447pite6ult8\"" Nov 03 16:12:11.202402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202389 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 16:12:11.202458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202409 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h7tfb\"" Nov 03 16:12:11.202497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202487 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 16:12:11.202620 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202605 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 16:12:11.202674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202618 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:12:11.202674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202609 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:12:11.202674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202662 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 16:12:11.202801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202675 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 16:12:11.202801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.202750 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 16:12:11.203158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203137 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6md\" (UniqueName: \"kubernetes.io/projected/4c0428d5-c0b3-45e7-8c92-9fdcdc798786-kube-api-access-jn6md\") pod \"tuned-54n6m\" (UID: \"4c0428d5-c0b3-45e7-8c92-9fdcdc798786\") " pod="openshift-cluster-node-tuning-operator/tuned-54n6m" Nov 03 16:12:11.203256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203165 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjnh\" (UniqueName: \"kubernetes.io/projected/8540df11-165c-4b01-b8a3-4042a8f6fdab-kube-api-access-6fjnh\") pod \"network-metrics-daemon-fddzx\" (UID: \"8540df11-165c-4b01-b8a3-4042a8f6fdab\") " pod="openshift-multus/network-metrics-daemon-fddzx" Nov 03 16:12:11.203256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203166 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6q9\" (UniqueName: \"kubernetes.io/projected/ecdd9d33-7747-417b-bff0-688639640a48-kube-api-access-kr6q9\") pod \"node-resolver-xt54q\" (UID: \"ecdd9d33-7747-417b-bff0-688639640a48\") " pod="openshift-dns/node-resolver-xt54q" Nov 03 16:12:11.203344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203271 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56sh\" (UniqueName: \"kubernetes.io/projected/289d2a22-071a-436a-afa6-8bfecabaa60c-kube-api-access-c56sh\") pod \"iptables-alerter-x86gk\" (UID: \"289d2a22-071a-436a-afa6-8bfecabaa60c\") " pod="openshift-network-operator/iptables-alerter-x86gk" Nov 03 16:12:11.203344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203278 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp64h\" (UniqueName: \"kubernetes.io/projected/507f7522-ec6f-4e48-b362-68e1c9dafeb8-kube-api-access-jp64h\") pod \"multus-additional-cni-plugins-kv69g\" (UID: \"507f7522-ec6f-4e48-b362-68e1c9dafeb8\") " pod="openshift-multus/multus-additional-cni-plugins-kv69g" Nov 03 16:12:11.203344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203271 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd77\" (UniqueName: \"kubernetes.io/projected/43230f8a-cf14-4872-b51a-a4f675f947fb-kube-api-access-qwd77\") pod \"ovnkube-node-gsdp4\" (UID: \"43230f8a-cf14-4872-b51a-a4f675f947fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:11.203636 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203618 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gvz\" (UniqueName: \"kubernetes.io/projected/caf14ab9-d061-4cfb-bc58-68ecf17a10a7-kube-api-access-42gvz\") pod \"network-check-target-67tfw\" (UID: \"caf14ab9-d061-4cfb-bc58-68ecf17a10a7\") " pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:12:11.203667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203656 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skprg\" (UniqueName: \"kubernetes.io/projected/61daa065-4042-47db-add5-514189226920-kube-api-access-skprg\") pod \"multus-8r945\" (UID: \"61daa065-4042-47db-add5-514189226920\") " pod="openshift-multus/multus-8r945" Nov 03 16:12:11.203720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203703 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrdq\" (UniqueName: \"kubernetes.io/projected/c0c67fab-535b-4a8e-a336-ca475185c3dd-kube-api-access-ltrdq\") pod \"node-ca-bcgxn\" (UID: \"c0c67fab-535b-4a8e-a336-ca475185c3dd\") " pod="openshift-image-registry/node-ca-bcgxn" Nov 03 16:12:11.203750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.203737 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7648n\" (UniqueName: \"kubernetes.io/projected/acc9fe0f-08e8-4cb1-bab7-ace0f47412c3-kube-api-access-7648n\") pod \"aws-ebs-csi-driver-node-pcfhd\" (UID: \"acc9fe0f-08e8-4cb1-bab7-ace0f47412c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcfhd" Nov 03 16:12:11.204811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.204790 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-secret-metrics-server-tls\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.204853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.204800 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.204853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.204821 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.204853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.204843 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cedb2971-5861-4832-bda4-242c206a0822-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:12:11.205933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.205916 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.206003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.205993 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.206058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.206013 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.206103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.206063 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.208077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.208057 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e29f1414-146f-4bb2-a62c-a29b6f54ff75-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.208244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.208230 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e29f1414-146f-4bb2-a62c-a29b6f54ff75-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.233265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.233240 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czjl\" (UniqueName: \"kubernetes.io/projected/7da5e2a9-8f58-4ee1-8a00-c443413a249c-kube-api-access-8czjl\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.233619 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.233599 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4nn\" (UniqueName: \"kubernetes.io/projected/5a8ec6fc-a61e-4d85-b9c8-f594647b0a96-kube-api-access-rx4nn\") pod \"metrics-server-d7dfbc995-gq2ct\" (UID: \"5a8ec6fc-a61e-4d85-b9c8-f594647b0a96\") " pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:11.233669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.233601 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxqv\" (UniqueName: \"kubernetes.io/projected/05b3881e-7078-461d-9c94-69d049ccc417-kube-api-access-8pxqv\") pod \"ingress-canary-wc6dg\" (UID: \"05b3881e-7078-461d-9c94-69d049ccc417\") " pod="openshift-ingress-canary/ingress-canary-wc6dg" Nov 03 16:12:11.233956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.233932 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvlq9\" (UniqueName: \"kubernetes.io/projected/61a83e02-ce4d-4df6-bc29-368ec906215f-kube-api-access-mvlq9\") pod \"kube-state-metrics-5dc8d7d89b-wfsql\" (UID: \"61a83e02-ce4d-4df6-bc29-368ec906215f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wfsql" Nov 03 16:12:11.234225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.234188 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h28\" (UniqueName: \"kubernetes.io/projected/cedb2971-5861-4832-bda4-242c206a0822-kube-api-access-r5h28\") pod \"monitoring-plugin-77f4f76594-nw79m\" (UID: \"cedb2971-5861-4832-bda4-242c206a0822\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:12:11.234334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.234318 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtmh\" (UniqueName: \"kubernetes.io/projected/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-kube-api-access-lhtmh\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.234702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.234684 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62d8\" (UniqueName: \"kubernetes.io/projected/2218179a-fa77-4676-ad59-885f522a5039-kube-api-access-j62d8\") pod \"dns-default-xg99b\" (UID: \"2218179a-fa77-4676-ad59-885f522a5039\") " pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:11.234879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.234862 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrll\" (UniqueName: \"kubernetes.io/projected/5e9344a7-9120-40ef-8671-5bc1fd4de206-kube-api-access-7xrll\") pod \"thanos-querier-5b684bf768-ghn5f\" (UID: \"5e9344a7-9120-40ef-8671-5bc1fd4de206\") " pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:11.235262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.235245 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fdm\" (UniqueName: \"kubernetes.io/projected/3833015d-9c9e-49d7-8630-a4b118d57ab6-kube-api-access-w5fdm\") pod \"openshift-state-metrics-7bd7489957-bcvnw\" (UID: \"3833015d-9c9e-49d7-8630-a4b118d57ab6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bcvnw" Nov 03 16:12:11.235308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.235288 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxqq\" (UniqueName: \"kubernetes.io/projected/e29f1414-146f-4bb2-a62c-a29b6f54ff75-kube-api-access-8xxqq\") pod \"prometheus-k8s-0\" (UID: \"e29f1414-146f-4bb2-a62c-a29b6f54ff75\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:11.278175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278153 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-sys\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.278266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278181 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-wtmp\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.278266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278196 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7da5e2a9-8f58-4ee1-8a00-c443413a249c-crio-socket\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.278266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278253 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-sys\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.278395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278272 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7da5e2a9-8f58-4ee1-8a00-c443413a249c-crio-socket\") pod \"insights-runtime-extractor-qbst4\" (UID: \"7da5e2a9-8f58-4ee1-8a00-c443413a249c\") " pod="openshift-insights/insights-runtime-extractor-qbst4" Nov 03 16:12:11.278395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278286 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-node-exporter-wtmp\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.278395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278317 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-root\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.278395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.278387 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/234cc11d-353c-4f0c-ab3b-4616ddd0b26d-root\") pod \"node-exporter-hfllm\" (UID: \"234cc11d-353c-4f0c-ab3b-4616ddd0b26d\") " pod="openshift-monitoring/node-exporter-hfllm" Nov 03 16:12:11.897391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:11.897366 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b684bf768-ghn5f" Nov 03 16:12:12.356321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:12.356295 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:12:12.584584 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:12.584549 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:12:14.301521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:14.301481 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xg99b" Nov 03 16:12:15.489869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:15.489842 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nw79m" Nov 03 16:12:15.888751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:15.888654 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:12:16.238393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:16.238368 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:19.133624 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:19.133592 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d7dfbc995-gq2ct" Nov 03 16:12:21.105481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:21.105453 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hm72k" Nov 03 16:12:21.209086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:21.209059 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:21.209276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:21.209264 5946 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:12:21.327597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:21.327565 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:21.515304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:21.515271 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:23.952887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:23.952853 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:27.105682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:27.105651 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsdp4" Nov 03 16:12:33.492444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:33.492406 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:33.685511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:33.685482 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:33.874816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:33.874728 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:34.072603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:34.072560 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:39.090535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:39.090496 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:39.279289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:39.279195 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:41.953359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:41.953321 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:42.144723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:42.144682 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:42.584506 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:12:42.584472 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:12:43.479310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:43.479278 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:43.674708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:43.674674 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:45.891398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:45.891319 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-67tfw" Nov 03 16:12:51.832562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:51.832524 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:52.020265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:52.020226 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:56.604892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:56.604859 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:56.793811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:12:56.793780 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:13:07.214527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:13:07.214493 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:13:12.583999 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:13:12.583957 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:13:24.325032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:13:24.324997 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:13:42.584377 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:13:42.584336 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:14:12.584563 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:14:12.584527 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:14:18.439756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:18.439705 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:19.392861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:19.392813 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:24.631690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:24.631660 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:32.322403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:32.322364 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:33.772988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.772957 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:33.928982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.928949 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf"] Nov 03 16:14:33.931515 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.931500 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:33.965217 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.965169 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:33.985637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.985603 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 16:14:33.989936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.989911 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:33.990034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.989959 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:33.990115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:33.990098 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7qx\" (UniqueName: \"kubernetes.io/projected/189aa2bc-b306-4a80-a225-7a4051d5f0fe-kube-api-access-nr7qx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.036716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.036633 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 16:14:34.036896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.036864 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mjwsb\"" Nov 03 16:14:34.037353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.037335 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf"] Nov 03 16:14:34.091163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.091132 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.091312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.091179 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.091312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.091235 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7qx\" (UniqueName: \"kubernetes.io/projected/189aa2bc-b306-4a80-a225-7a4051d5f0fe-kube-api-access-nr7qx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.091610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.091589 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.091649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.091598 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.149974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.149942 5946 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:14:34.152346 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.152316 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7qx\" (UniqueName: \"kubernetes.io/projected/189aa2bc-b306-4a80-a225-7a4051d5f0fe-kube-api-access-nr7qx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.243921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.243884 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:14:34.619187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.619118 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf"] Nov 03 16:14:34.622261 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:14:34.622233 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189aa2bc_b306_4a80_a225_7a4051d5f0fe.slice/crio-086b2fd8b156e5f0e9f3264570c56cfe1a93dd3d07ffc01b1dda011182860fb6 WatchSource:0}: Error finding container 086b2fd8b156e5f0e9f3264570c56cfe1a93dd3d07ffc01b1dda011182860fb6: Status 404 returned error can't find the container with id 086b2fd8b156e5f0e9f3264570c56cfe1a93dd3d07ffc01b1dda011182860fb6 Nov 03 16:14:34.624337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:34.624322 5946 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:14:35.089754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:35.089727 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:35.558338 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:35.558303 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" event={"ID":"189aa2bc-b306-4a80-a225-7a4051d5f0fe","Type":"ContainerStarted","Data":"086b2fd8b156e5f0e9f3264570c56cfe1a93dd3d07ffc01b1dda011182860fb6"} Nov 03 16:14:36.141799 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.141768 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d"] Nov 03 16:14:36.143897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.143882 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.207767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.207516 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.207962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.207872 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqt2\" (UniqueName: \"kubernetes.io/projected/be8319f1-c204-4eb5-b96a-74f58b7a6e93-kube-api-access-ctqt2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.208134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.208086 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.249382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.249348 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-66g4h\"" Nov 03 16:14:36.249946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.249927 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d"] Nov 03 16:14:36.308782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.308750 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.308946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.308806 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.308946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.308828 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqt2\" (UniqueName: \"kubernetes.io/projected/be8319f1-c204-4eb5-b96a-74f58b7a6e93-kube-api-access-ctqt2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.309152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.309134 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.309194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.309165 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.371199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.371162 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqt2\" (UniqueName: \"kubernetes.io/projected/be8319f1-c204-4eb5-b96a-74f58b7a6e93-kube-api-access-ctqt2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.456415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.456328 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:14:36.738859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.738831 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g"] Nov 03 16:14:36.741360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.741344 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.796778 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.796744 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-m8skf\"" Nov 03 16:14:36.804803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.804774 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g"] Nov 03 16:14:36.813661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.813628 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk85k\" (UniqueName: \"kubernetes.io/projected/7d72eca4-7c09-49f6-9856-febd8b8fab31-kube-api-access-qk85k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.813815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.813702 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.813815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.813798 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.847027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.846997 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g"] Nov 03 16:14:36.849601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.849581 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:36.871194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.871170 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d"] Nov 03 16:14:36.895903 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:14:36.895873 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8319f1_c204_4eb5_b96a_74f58b7a6e93.slice/crio-618c5abf3f19d967fa0e2c9a4c64b8e80965762d1041b51a2106de6633dce8b8 WatchSource:0}: Error finding container 618c5abf3f19d967fa0e2c9a4c64b8e80965762d1041b51a2106de6633dce8b8: Status 404 returned error can't find the container with id 618c5abf3f19d967fa0e2c9a4c64b8e80965762d1041b51a2106de6633dce8b8 Nov 03 16:14:36.905321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.905289 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9kznv\"" Nov 03 16:14:36.914720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.914695 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:36.914839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.914739 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-kube-api-access-d2wpm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:36.914901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.914844 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:36.914962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.914937 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk85k\" (UniqueName: \"kubernetes.io/projected/7d72eca4-7c09-49f6-9856-febd8b8fab31-kube-api-access-qk85k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.915020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.914996 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.915122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.915020 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.915411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.915394 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.915472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.915452 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:36.917888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.917808 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g"] Nov 03 16:14:36.951002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.950968 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f"] Nov 03 16:14:36.954064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.954043 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:36.974282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:36.974262 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk85k\" (UniqueName: \"kubernetes.io/projected/7d72eca4-7c09-49f6-9856-febd8b8fab31-kube-api-access-qk85k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:37.009115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.009042 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-nfbkg\"" Nov 03 16:14:37.014354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.014326 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f"] Nov 03 16:14:37.015750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.015732 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6ks\" (UniqueName: \"kubernetes.io/projected/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-kube-api-access-2b6ks\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.015815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.015762 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.015815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.015793 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:37.015898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.015862 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-kube-api-access-d2wpm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:37.015898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.015890 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:37.016025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.016014 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.016161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.016141 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:37.016237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.016199 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:37.054700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.054674 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:14:37.076434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.076397 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-kube-api-access-d2wpm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:37.117302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.117267 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.117444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.117355 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6ks\" (UniqueName: \"kubernetes.io/projected/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-kube-api-access-2b6ks\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.117444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.117392 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.117766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.117738 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.117766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.117754 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.163864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.163830 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:14:37.177830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.177786 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6ks\" (UniqueName: \"kubernetes.io/projected/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-kube-api-access-2b6ks\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.266776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.266697 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:14:37.432579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.432500 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g"] Nov 03 16:14:37.436982 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:14:37.436950 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d72eca4_7c09_49f6_9856_febd8b8fab31.slice/crio-395cf55c757ed5b23df1f6794e1ebc423e978eda4d51dbf00dc317df741c7ab2 WatchSource:0}: Error finding container 395cf55c757ed5b23df1f6794e1ebc423e978eda4d51dbf00dc317df741c7ab2: Status 404 returned error can't find the container with id 395cf55c757ed5b23df1f6794e1ebc423e978eda4d51dbf00dc317df741c7ab2 Nov 03 16:14:37.548893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.548803 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g"] Nov 03 16:14:37.552041 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:14:37.552010 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa1d860_c5a0_4d77_a5e7_b7bb7b3eb922.slice/crio-a356d21405c860cbff03179e266f922fa384ec51666f446ddd3758f098662a41 WatchSource:0}: Error finding container a356d21405c860cbff03179e266f922fa384ec51666f446ddd3758f098662a41: Status 404 returned error can't find the container with id a356d21405c860cbff03179e266f922fa384ec51666f446ddd3758f098662a41 Nov 03 16:14:37.565977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.565948 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" event={"ID":"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922","Type":"ContainerStarted","Data":"a356d21405c860cbff03179e266f922fa384ec51666f446ddd3758f098662a41"} Nov 03 16:14:37.567044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.567023 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" event={"ID":"7d72eca4-7c09-49f6-9856-febd8b8fab31","Type":"ContainerStarted","Data":"395cf55c757ed5b23df1f6794e1ebc423e978eda4d51dbf00dc317df741c7ab2"} Nov 03 16:14:37.568178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.568158 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" event={"ID":"be8319f1-c204-4eb5-b96a-74f58b7a6e93","Type":"ContainerStarted","Data":"618c5abf3f19d967fa0e2c9a4c64b8e80965762d1041b51a2106de6633dce8b8"} Nov 03 16:14:37.637780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:37.637747 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f"] Nov 03 16:14:37.641729 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:14:37.641696 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe89bec_8cbb_4cb8_822b_02e55bfb0257.slice/crio-d929b19277fc4dd7d797a610a40041f7775214e1f4c8d5e66416d49d95a8c1f4 WatchSource:0}: Error finding container d929b19277fc4dd7d797a610a40041f7775214e1f4c8d5e66416d49d95a8c1f4: Status 404 returned error can't find the container with id d929b19277fc4dd7d797a610a40041f7775214e1f4c8d5e66416d49d95a8c1f4 Nov 03 16:14:38.602154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:38.602116 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" event={"ID":"2fe89bec-8cbb-4cb8-822b-02e55bfb0257","Type":"ContainerStarted","Data":"d929b19277fc4dd7d797a610a40041f7775214e1f4c8d5e66416d49d95a8c1f4"} Nov 03 16:14:41.757412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:41.757370 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:41.968946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:41.968904 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:42.589808 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:14:42.589774 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:14:43.199535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.199496 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:43.368346 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.368308 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd"] Nov 03 16:14:43.371030 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.371005 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.427982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.427948 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 16:14:43.427982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.427948 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-fvh28\"" Nov 03 16:14:43.478765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.478678 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.478765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.478728 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.478957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.478840 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.478957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.478879 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.478957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.478910 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.479073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.479035 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.479114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.479085 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e8e85e-5124-4a7e-85e5-7373fba14616-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.479228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.479149 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5tn\" (UniqueName: \"kubernetes.io/projected/77e8e85e-5124-4a7e-85e5-7373fba14616-kube-api-access-md5tn\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.479228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.479181 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.479727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.479698 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd"] Nov 03 16:14:43.519183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.519113 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:43.527912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.527883 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 16:14:43.528042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.527995 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 16:14:43.580275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580240 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580283 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580307 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580370 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580415 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e8e85e-5124-4a7e-85e5-7373fba14616-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580440 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5tn\" (UniqueName: \"kubernetes.io/projected/77e8e85e-5124-4a7e-85e5-7373fba14616-kube-api-access-md5tn\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580471 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580512 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580539 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580837 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.580921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580892 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.581016 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.580990 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.581312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.581249 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.581428 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.581345 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e8e85e-5124-4a7e-85e5-7373fba14616-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.583068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.583043 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.583686 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.583666 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.643734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.643699 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e8e85e-5124-4a7e-85e5-7373fba14616-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.644282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.644236 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5tn\" (UniqueName: \"kubernetes.io/projected/77e8e85e-5124-4a7e-85e5-7373fba14616-kube-api-access-md5tn\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd\" (UID: \"77e8e85e-5124-4a7e-85e5-7373fba14616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.687941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.687909 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:43.707730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:43.707669 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:44.128489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:44.128318 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd"] Nov 03 16:14:44.133018 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:14:44.132938 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e8e85e_5124_4a7e_85e5_7373fba14616.slice/crio-b538db7f9b8081ba8cef9d5ecdc66990d679d5002f2f39458926a385386e2ed3 WatchSource:0}: Error finding container b538db7f9b8081ba8cef9d5ecdc66990d679d5002f2f39458926a385386e2ed3: Status 404 returned error can't find the container with id b538db7f9b8081ba8cef9d5ecdc66990d679d5002f2f39458926a385386e2ed3 Nov 03 16:14:44.638460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:44.638424 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" event={"ID":"77e8e85e-5124-4a7e-85e5-7373fba14616","Type":"ContainerStarted","Data":"b538db7f9b8081ba8cef9d5ecdc66990d679d5002f2f39458926a385386e2ed3"} Nov 03 16:14:47.202648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:47.202618 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:47.660881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:47.660844 5946 generic.go:356] "Generic (PLEG): container finished" podID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerID="e5b36db914619be74c2ffa9ef42de1c5f37cd7f68d97087b92bd2fd7eafda14a" exitCode=0 Nov 03 16:14:47.661145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:47.661118 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" event={"ID":"2fe89bec-8cbb-4cb8-822b-02e55bfb0257","Type":"ContainerDied","Data":"e5b36db914619be74c2ffa9ef42de1c5f37cd7f68d97087b92bd2fd7eafda14a"} Nov 03 16:14:47.663337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:47.663308 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" event={"ID":"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922","Type":"ContainerStarted","Data":"e2f0366ad4951ce2685d103a3cb69248f7a6d24b168bfa1c8ee440248581a4f4"} Nov 03 16:14:47.665608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:47.665583 5946 generic.go:356] "Generic (PLEG): container finished" podID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerID="8e87086c31540dfd95c4bb7071469aeb117d511631927423d0df4b7d0a8bf55f" exitCode=0 Nov 03 16:14:47.665710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:47.665670 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" event={"ID":"7d72eca4-7c09-49f6-9856-febd8b8fab31","Type":"ContainerDied","Data":"8e87086c31540dfd95c4bb7071469aeb117d511631927423d0df4b7d0a8bf55f"} Nov 03 16:14:48.671556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:48.671516 5946 generic.go:356] "Generic (PLEG): container finished" podID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerID="e2f0366ad4951ce2685d103a3cb69248f7a6d24b168bfa1c8ee440248581a4f4" exitCode=0 Nov 03 16:14:48.672013 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:48.671599 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" event={"ID":"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922","Type":"ContainerDied","Data":"e2f0366ad4951ce2685d103a3cb69248f7a6d24b168bfa1c8ee440248581a4f4"} Nov 03 16:14:48.674089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:48.674065 5946 generic.go:356] "Generic (PLEG): container finished" podID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerID="4ec65b7753ffa0aa05eeca5232660abfacbbef251e42e0e1899ebe617f59539e" exitCode=0 Nov 03 16:14:48.674199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:48.674164 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" event={"ID":"be8319f1-c204-4eb5-b96a-74f58b7a6e93","Type":"ContainerDied","Data":"4ec65b7753ffa0aa05eeca5232660abfacbbef251e42e0e1899ebe617f59539e"} Nov 03 16:14:48.676439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:48.676378 5946 generic.go:356] "Generic (PLEG): container finished" podID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerID="af970e45a05f8d811da093155b7829d8301d2ea83033cb98c0e10e8c21eb9865" exitCode=0 Nov 03 16:14:48.676439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:48.676413 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" event={"ID":"189aa2bc-b306-4a80-a225-7a4051d5f0fe","Type":"ContainerDied","Data":"af970e45a05f8d811da093155b7829d8301d2ea83033cb98c0e10e8c21eb9865"} Nov 03 16:14:49.500981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.500949 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:14:49.501072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.501021 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:14:49.501072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.501049 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:14:49.682957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.682885 5946 generic.go:356] "Generic (PLEG): container finished" podID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerID="1c20154d3eed38e320da4124f434502ed3044c12b18f51f4e2158ee7756e0d37" exitCode=0 Nov 03 16:14:49.683408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.682973 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" event={"ID":"7d72eca4-7c09-49f6-9856-febd8b8fab31","Type":"ContainerDied","Data":"1c20154d3eed38e320da4124f434502ed3044c12b18f51f4e2158ee7756e0d37"} Nov 03 16:14:49.684756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.684731 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" event={"ID":"77e8e85e-5124-4a7e-85e5-7373fba14616","Type":"ContainerStarted","Data":"0c6e6c351f9a03eb089ed0d9034cb96e1edf06a037ba63886d86065e588c7b48"} Nov 03 16:14:49.687319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.687298 5946 generic.go:356] "Generic (PLEG): container finished" podID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerID="09b0f1255598767f2ef4d081dee941704c141893e9c5e90e9066db8bacbdb729" exitCode=0 Nov 03 16:14:49.687436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.687353 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" event={"ID":"2fe89bec-8cbb-4cb8-822b-02e55bfb0257","Type":"ContainerDied","Data":"09b0f1255598767f2ef4d081dee941704c141893e9c5e90e9066db8bacbdb729"} Nov 03 16:14:49.688233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:49.688201 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:50.062063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:50.062001 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" podStartSLOduration=1.697565846 podStartE2EDuration="7.061982075s" totalImagesPullingTime="5.364416229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:50.061548845 +0000 UTC m=+160.182601667" watchObservedRunningTime="2026-11-03 16:14:50.061982075 +0000 UTC m=+160.183034897" Nov 03 16:14:50.693727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:50.693698 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:50.694444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:50.694410 5946 generic.go:356] "Generic (PLEG): container finished" podID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerID="39965a24b59fad64f3bd04330e97d8b034d7b3f72789d0b2a46a2e178b82c76a" exitCode=0 Nov 03 16:14:50.694581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:50.694494 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" event={"ID":"189aa2bc-b306-4a80-a225-7a4051d5f0fe","Type":"ContainerDied","Data":"39965a24b59fad64f3bd04330e97d8b034d7b3f72789d0b2a46a2e178b82c76a"} Nov 03 16:14:50.694790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:50.694773 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:50.695694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:50.695627 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd" Nov 03 16:14:51.700947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:51.700911 5946 generic.go:356] "Generic (PLEG): container finished" podID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerID="45081ad7c106f8fbe04acb20934b6b24c5924ad74ec27041f203e0e9e7318339" exitCode=0 Nov 03 16:14:51.701487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:51.701003 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" event={"ID":"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922","Type":"ContainerDied","Data":"45081ad7c106f8fbe04acb20934b6b24c5924ad74ec27041f203e0e9e7318339"} Nov 03 16:14:54.721480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:54.721444 5946 generic.go:356] "Generic (PLEG): container finished" podID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerID="1a53d781da318a0bcd22d40807e6b5c93cd9d377ee30c755c1e3b6d7f1b7bfa9" exitCode=0 Nov 03 16:14:54.722125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:54.721557 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" event={"ID":"be8319f1-c204-4eb5-b96a-74f58b7a6e93","Type":"ContainerDied","Data":"1a53d781da318a0bcd22d40807e6b5c93cd9d377ee30c755c1e3b6d7f1b7bfa9"} Nov 03 16:14:59.754799 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:59.754761 5946 generic.go:356] "Generic (PLEG): container finished" podID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerID="05f09014c5a7db3df463e3ee955bbccba5afc0f931382dc18557775e6536276e" exitCode=0 Nov 03 16:14:59.755138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:14:59.754874 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" event={"ID":"7d72eca4-7c09-49f6-9856-febd8b8fab31","Type":"ContainerDied","Data":"05f09014c5a7db3df463e3ee955bbccba5afc0f931382dc18557775e6536276e"} Nov 03 16:15:00.761846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:00.761807 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" event={"ID":"2fe89bec-8cbb-4cb8-822b-02e55bfb0257","Type":"ContainerStarted","Data":"e61a174524f963a02ee3f80882428705704b13325819946432f07f13919f66bf"} Nov 03 16:15:00.983406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:00.983373 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:15:01.066815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.066784 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d72eca4-7c09-49f6-9856-febd8b8fab31-kube-api-access-qk85k\" (UniqueName: \"kubernetes.io/projected/7d72eca4-7c09-49f6-9856-febd8b8fab31-kube-api-access-qk85k\") pod \"7d72eca4-7c09-49f6-9856-febd8b8fab31\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " Nov 03 16:15:01.066948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.066833 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-util\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-util\") pod \"7d72eca4-7c09-49f6-9856-febd8b8fab31\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " Nov 03 16:15:01.066948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.066866 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-bundle\") pod \"7d72eca4-7c09-49f6-9856-febd8b8fab31\" (UID: \"7d72eca4-7c09-49f6-9856-febd8b8fab31\") " Nov 03 16:15:01.067482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.067429 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-bundle" pod "7d72eca4-7c09-49f6-9856-febd8b8fab31" (UID: "7d72eca4-7c09-49f6-9856-febd8b8fab31"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:01.069591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.069565 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d72eca4-7c09-49f6-9856-febd8b8fab31-kube-api-access-qk85k" pod "7d72eca4-7c09-49f6-9856-febd8b8fab31" (UID: "7d72eca4-7c09-49f6-9856-febd8b8fab31"). InnerVolumeSpecName "kube-api-access-qk85k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:01.071687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.071668 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-util" pod "7d72eca4-7c09-49f6-9856-febd8b8fab31" (UID: "7d72eca4-7c09-49f6-9856-febd8b8fab31"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:01.124467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.124394 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" podStartSLOduration=2.902854599 podStartE2EDuration="25.124373637s" totalImagesPullingTime="22.221519038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:00.904489599 +0000 UTC m=+171.025542421" watchObservedRunningTime="2026-11-03 16:15:01.124373637 +0000 UTC m=+171.245426462" Nov 03 16:15:01.167783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.167744 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk85k\" (UniqueName: \"kubernetes.io/projected/7d72eca4-7c09-49f6-9856-febd8b8fab31-kube-api-access-qk85k\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:01.167783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.167779 5946 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-util\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:01.168047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.167795 5946 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d72eca4-7c09-49f6-9856-febd8b8fab31-bundle\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:01.767498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.767464 5946 generic.go:356] "Generic (PLEG): container finished" podID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerID="e61a174524f963a02ee3f80882428705704b13325819946432f07f13919f66bf" exitCode=0 Nov 03 16:15:01.767949 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.767553 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" event={"ID":"2fe89bec-8cbb-4cb8-822b-02e55bfb0257","Type":"ContainerDied","Data":"e61a174524f963a02ee3f80882428705704b13325819946432f07f13919f66bf"} Nov 03 16:15:01.769465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.769445 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" Nov 03 16:15:01.769606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.769448 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68xs9g" event={"ID":"7d72eca4-7c09-49f6-9856-febd8b8fab31","Type":"ContainerDied","Data":"395cf55c757ed5b23df1f6794e1ebc423e978eda4d51dbf00dc317df741c7ab2"} Nov 03 16:15:01.769606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.769557 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395cf55c757ed5b23df1f6794e1ebc423e978eda4d51dbf00dc317df741c7ab2" Nov 03 16:15:01.771657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.771633 5946 generic.go:356] "Generic (PLEG): container finished" podID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerID="9ca4cff1d06dffcf5213eca82f26cb7a308fd3b129f890dfc6814c7d0f595ab4" exitCode=0 Nov 03 16:15:01.771758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:01.771702 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" event={"ID":"189aa2bc-b306-4a80-a225-7a4051d5f0fe","Type":"ContainerDied","Data":"9ca4cff1d06dffcf5213eca82f26cb7a308fd3b129f890dfc6814c7d0f595ab4"} Nov 03 16:15:02.778061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.778029 5946 generic.go:356] "Generic (PLEG): container finished" podID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerID="7b62035d72a9edd5f16a75c2464f9901f1d5912e18ba43a5a849f42b5f87c0b4" exitCode=0 Nov 03 16:15:02.778507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.778124 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" event={"ID":"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922","Type":"ContainerDied","Data":"7b62035d72a9edd5f16a75c2464f9901f1d5912e18ba43a5a849f42b5f87c0b4"} Nov 03 16:15:02.780405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.780377 5946 generic.go:356] "Generic (PLEG): container finished" podID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerID="cb8e83092c4cd6709012d700e428ecc44eb44012590c5a40f8f561fc8aafd6c0" exitCode=0 Nov 03 16:15:02.780532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.780458 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" event={"ID":"be8319f1-c204-4eb5-b96a-74f58b7a6e93","Type":"ContainerDied","Data":"cb8e83092c4cd6709012d700e428ecc44eb44012590c5a40f8f561fc8aafd6c0"} Nov 03 16:15:02.926521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.926488 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:15:02.948867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.948842 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:15:02.989048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.989016 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-bundle\") pod \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " Nov 03 16:15:02.989198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.989097 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-util\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-util\") pod \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " Nov 03 16:15:02.989198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.989140 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-kube-api-access-2b6ks\" (UniqueName: \"kubernetes.io/projected/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-kube-api-access-2b6ks\") pod \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " Nov 03 16:15:02.989198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.989173 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189aa2bc-b306-4a80-a225-7a4051d5f0fe-kube-api-access-nr7qx\" (UniqueName: \"kubernetes.io/projected/189aa2bc-b306-4a80-a225-7a4051d5f0fe-kube-api-access-nr7qx\") pod \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " Nov 03 16:15:02.989412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.989245 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-bundle\") pod \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\" (UID: \"189aa2bc-b306-4a80-a225-7a4051d5f0fe\") " Nov 03 16:15:02.989412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.989277 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-util\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-util\") pod \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\" (UID: \"2fe89bec-8cbb-4cb8-822b-02e55bfb0257\") " Nov 03 16:15:02.989949 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.989900 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-bundle" pod "2fe89bec-8cbb-4cb8-822b-02e55bfb0257" (UID: "2fe89bec-8cbb-4cb8-822b-02e55bfb0257"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:02.990122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.990089 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-bundle" pod "189aa2bc-b306-4a80-a225-7a4051d5f0fe" (UID: "189aa2bc-b306-4a80-a225-7a4051d5f0fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:02.991465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.991436 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-kube-api-access-2b6ks" pod "2fe89bec-8cbb-4cb8-822b-02e55bfb0257" (UID: "2fe89bec-8cbb-4cb8-822b-02e55bfb0257"). InnerVolumeSpecName "kube-api-access-2b6ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:02.991734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.991716 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189aa2bc-b306-4a80-a225-7a4051d5f0fe-kube-api-access-nr7qx" pod "189aa2bc-b306-4a80-a225-7a4051d5f0fe" (UID: "189aa2bc-b306-4a80-a225-7a4051d5f0fe"). InnerVolumeSpecName "kube-api-access-nr7qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:02.993484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.993460 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-util" pod "2fe89bec-8cbb-4cb8-822b-02e55bfb0257" (UID: "2fe89bec-8cbb-4cb8-822b-02e55bfb0257"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:02.994971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:02.994949 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-util" pod "189aa2bc-b306-4a80-a225-7a4051d5f0fe" (UID: "189aa2bc-b306-4a80-a225-7a4051d5f0fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:03.089990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.089956 5946 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-bundle\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:03.089990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.089989 5946 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-util\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:03.089990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.090001 5946 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-bundle\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:03.090228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.090011 5946 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/189aa2bc-b306-4a80-a225-7a4051d5f0fe-util\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:03.090228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.090021 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b6ks\" (UniqueName: \"kubernetes.io/projected/2fe89bec-8cbb-4cb8-822b-02e55bfb0257-kube-api-access-2b6ks\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:03.090228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.090031 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7qx\" (UniqueName: \"kubernetes.io/projected/189aa2bc-b306-4a80-a225-7a4051d5f0fe-kube-api-access-nr7qx\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:03.788921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.788891 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" Nov 03 16:15:03.788921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.788907 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcf6f" event={"ID":"2fe89bec-8cbb-4cb8-822b-02e55bfb0257","Type":"ContainerDied","Data":"d929b19277fc4dd7d797a610a40041f7775214e1f4c8d5e66416d49d95a8c1f4"} Nov 03 16:15:03.789461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.788957 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d929b19277fc4dd7d797a610a40041f7775214e1f4c8d5e66416d49d95a8c1f4" Nov 03 16:15:03.791071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.791054 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" Nov 03 16:15:03.791199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.791091 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kkwgf" event={"ID":"189aa2bc-b306-4a80-a225-7a4051d5f0fe","Type":"ContainerDied","Data":"086b2fd8b156e5f0e9f3264570c56cfe1a93dd3d07ffc01b1dda011182860fb6"} Nov 03 16:15:03.791199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.791138 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="086b2fd8b156e5f0e9f3264570c56cfe1a93dd3d07ffc01b1dda011182860fb6" Nov 03 16:15:03.972432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.972405 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:15:03.986307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.986283 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:15:03.997732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.997714 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-util\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-util\") pod \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " Nov 03 16:15:03.997795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.997748 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-kube-api-access-d2wpm\") pod \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " Nov 03 16:15:03.997795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.997779 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-bundle\") pod \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " Nov 03 16:15:03.997906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.997812 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-bundle\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-bundle\") pod \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " Nov 03 16:15:03.997960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.997928 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-util\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-util\") pod \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\" (UID: \"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922\") " Nov 03 16:15:03.998015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.997986 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8319f1-c204-4eb5-b96a-74f58b7a6e93-kube-api-access-ctqt2\" (UniqueName: \"kubernetes.io/projected/be8319f1-c204-4eb5-b96a-74f58b7a6e93-kube-api-access-ctqt2\") pod \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\" (UID: \"be8319f1-c204-4eb5-b96a-74f58b7a6e93\") " Nov 03 16:15:03.998377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.998349 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-bundle" pod "0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" (UID: "0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:03.998659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:03.998607 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-bundle" pod "be8319f1-c204-4eb5-b96a-74f58b7a6e93" (UID: "be8319f1-c204-4eb5-b96a-74f58b7a6e93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:04.000370 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.000350 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-kube-api-access-d2wpm" pod "0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" (UID: "0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922"). InnerVolumeSpecName "kube-api-access-d2wpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:04.001067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.001042 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8319f1-c204-4eb5-b96a-74f58b7a6e93-kube-api-access-ctqt2" pod "be8319f1-c204-4eb5-b96a-74f58b7a6e93" (UID: "be8319f1-c204-4eb5-b96a-74f58b7a6e93"). InnerVolumeSpecName "kube-api-access-ctqt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:04.005632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.005605 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-util" pod "be8319f1-c204-4eb5-b96a-74f58b7a6e93" (UID: "be8319f1-c204-4eb5-b96a-74f58b7a6e93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:04.006040 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.006023 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-util" pod "0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" (UID: "0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:04.099382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.099351 5946 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-bundle\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:04.099382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.099376 5946 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-bundle\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:04.099382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.099386 5946 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-util\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:04.099696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.099396 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqt2\" (UniqueName: \"kubernetes.io/projected/be8319f1-c204-4eb5-b96a-74f58b7a6e93-kube-api-access-ctqt2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:04.099696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.099406 5946 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be8319f1-c204-4eb5-b96a-74f58b7a6e93-util\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:04.099696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.099415 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922-kube-api-access-d2wpm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:15:04.797083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.797052 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" event={"ID":"0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922","Type":"ContainerDied","Data":"a356d21405c860cbff03179e266f922fa384ec51666f446ddd3758f098662a41"} Nov 03 16:15:04.797535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.797102 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a356d21405c860cbff03179e266f922fa384ec51666f446ddd3758f098662a41" Nov 03 16:15:04.797535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.797063 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hrg7g" Nov 03 16:15:04.799035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.799009 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" event={"ID":"be8319f1-c204-4eb5-b96a-74f58b7a6e93","Type":"ContainerDied","Data":"618c5abf3f19d967fa0e2c9a4c64b8e80965762d1041b51a2106de6633dce8b8"} Nov 03 16:15:04.799135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.799040 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnfc8d" Nov 03 16:15:04.799135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:04.799047 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="618c5abf3f19d967fa0e2c9a4c64b8e80965762d1041b51a2106de6633dce8b8" Nov 03 16:15:07.770811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:07.770745 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:08.028574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:08.028470 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:08.393551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:08.393458 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:08.742569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:08.742533 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:09.004366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.004273 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:09.004843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.004824 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg"] Nov 03 16:15:09.005131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005118 5946 cpu_manager.go:414] "containerMap: removing container" podUID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerName="util" Nov 03 16:15:09.005167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005136 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerName="util" Nov 03 16:15:09.005167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005151 5946 cpu_manager.go:414] "containerMap: removing container" podUID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerName="util" Nov 03 16:15:09.005167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005155 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerName="util" Nov 03 16:15:09.005167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005161 5946 cpu_manager.go:414] "containerMap: removing container" podUID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerName="extract" Nov 03 16:15:09.005167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005166 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005174 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005178 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005187 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005191 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005199 5946 cpu_manager.go:414] "containerMap: removing container" podUID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005220 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005230 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005235 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005241 5946 cpu_manager.go:414] "containerMap: removing container" podUID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005245 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005256 5946 cpu_manager.go:414] "containerMap: removing container" podUID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005260 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005265 5946 cpu_manager.go:414] "containerMap: removing container" podUID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005269 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005280 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerName="util" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005285 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerName="util" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005295 5946 cpu_manager.go:414] "containerMap: removing container" podUID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005302 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerName="extract" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005309 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerName="util" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005313 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerName="util" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005320 5946 cpu_manager.go:414] "containerMap: removing container" podUID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerName="util" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005323 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerName="util" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005329 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerName="pull" Nov 03 16:15:09.005345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005334 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerName="pull" Nov 03 16:15:09.006059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005394 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa1d860-c5a0-4d77-a5e7-b7bb7b3eb922" containerName="extract" Nov 03 16:15:09.006059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005404 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8319f1-c204-4eb5-b96a-74f58b7a6e93" containerName="extract" Nov 03 16:15:09.006059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005412 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="189aa2bc-b306-4a80-a225-7a4051d5f0fe" containerName="extract" Nov 03 16:15:09.006059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005417 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe89bec-8cbb-4cb8-822b-02e55bfb0257" containerName="extract" Nov 03 16:15:09.006059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.005424 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d72eca4-7c09-49f6-9856-febd8b8fab31" containerName="extract" Nov 03 16:15:09.007979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.007965 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.037327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.037291 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-tmp\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.037517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.037329 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gpg\" (UniqueName: \"kubernetes.io/projected/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-kube-api-access-z5gpg\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.037517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.037394 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.037644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.037518 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-manifests\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.037644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.037545 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.037644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.037573 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.100988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.100949 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 16:15:09.101154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.101003 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 16:15:09.113861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.113728 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 16:15:09.113861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.113755 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 16:15:09.113861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.113795 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rbhfv\"" Nov 03 16:15:09.113861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.113828 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 16:15:09.119537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.119510 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg"] Nov 03 16:15:09.138353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.138330 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-tmp\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.138505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.138358 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gpg\" (UniqueName: \"kubernetes.io/projected/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-kube-api-access-z5gpg\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.138505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.138391 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.138505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.138472 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-manifests\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.138505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.138496 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.138711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.138518 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.138865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.138841 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-manifests\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.141128 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.141101 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.141262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.141175 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.141262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.141176 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.141357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.141277 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-tmp\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.198908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.198881 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gpg\" (UniqueName: \"kubernetes.io/projected/28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0-kube-api-access-z5gpg\") pod \"opendatahub-operator-controller-manager-7d94989496-d9njg\" (UID: \"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.214398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.214358 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:09.319719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.319637 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:09.415449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.415412 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:09.665803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.665730 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:09.846087 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:09.846060 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bf6f7f_39ff_41a8_a1d1_47c7a65ac2b0.slice/crio-cb7a04c422a553969e00914b83b0f266594bec1c6b2f2fd5e9dd0c56ac6a413b WatchSource:0}: Error finding container cb7a04c422a553969e00914b83b0f266594bec1c6b2f2fd5e9dd0c56ac6a413b: Status 404 returned error can't find the container with id cb7a04c422a553969e00914b83b0f266594bec1c6b2f2fd5e9dd0c56ac6a413b Nov 03 16:15:09.893499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:09.893468 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg"] Nov 03 16:15:10.824068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:10.824025 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" event={"ID":"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0","Type":"ContainerStarted","Data":"cb7a04c422a553969e00914b83b0f266594bec1c6b2f2fd5e9dd0c56ac6a413b"} Nov 03 16:15:11.785056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:11.785018 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:12.404921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:12.404883 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:12.588954 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:15:12.588921 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:15:12.630667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:12.630625 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:15.849516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:15.849481 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" event={"ID":"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0","Type":"ContainerStarted","Data":"740b999cf224dab8f286c9e91ab9ac25a27036094b3d38ac6f66cd8b052592dc"} Nov 03 16:15:16.854917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:16.854883 5946 generic.go:356] "Generic (PLEG): container finished" podID="28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0" containerID="740b999cf224dab8f286c9e91ab9ac25a27036094b3d38ac6f66cd8b052592dc" exitCode=0 Nov 03 16:15:16.855324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:16.854923 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" event={"ID":"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0","Type":"ContainerDied","Data":"740b999cf224dab8f286c9e91ab9ac25a27036094b3d38ac6f66cd8b052592dc"} Nov 03 16:15:17.266344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:17.266304 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:17.497809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:17.497773 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:17.768964 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:17.768934 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:18.289887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:18.289840 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:18.864928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:18.864880 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" event={"ID":"28bf6f7f-39ff-41a8-a1d1-47c7a65ac2b0","Type":"ContainerStarted","Data":"25d0c93c7a470976979df27c129fb3413fcd08f92ac81e9512105f831df62351"} Nov 03 16:15:18.865097 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:18.865031 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:19.163350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:19.163248 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" podStartSLOduration=3.878332975 podStartE2EDuration="11.163232511s" totalImagesPullingTime="6.869700287s" totalInitContainerRuntime="415.199249ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:19.111825926 +0000 UTC m=+189.232878750" watchObservedRunningTime="2026-11-03 16:15:19.163232511 +0000 UTC m=+189.284285327" Nov 03 16:15:19.557242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:19.557189 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:21.553602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:21.553563 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:23.100268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.100235 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:23.296719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.296688 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:23.838270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.838237 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv"] Nov 03 16:15:23.842357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.842340 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:23.948467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.948438 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8ts2t\"" Nov 03 16:15:23.948635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.948491 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 16:15:23.948635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.948499 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 16:15:23.949576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.949557 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv"] Nov 03 16:15:23.963501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.963468 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brts8\" (UniqueName: \"kubernetes.io/projected/b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63-kube-api-access-brts8\") pod \"cert-manager-webhook-5dbbbf75cc-ns7gv\" (UID: \"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:23.963628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:23.963517 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ns7gv\" (UID: \"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:24.064420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:24.064387 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brts8\" (UniqueName: \"kubernetes.io/projected/b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63-kube-api-access-brts8\") pod \"cert-manager-webhook-5dbbbf75cc-ns7gv\" (UID: \"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:24.064597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:24.064430 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ns7gv\" (UID: \"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:24.126349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:24.126276 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ns7gv\" (UID: \"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:24.126680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:24.126387 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brts8\" (UniqueName: \"kubernetes.io/projected/b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63-kube-api-access-brts8\") pod \"cert-manager-webhook-5dbbbf75cc-ns7gv\" (UID: \"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:24.165171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:24.165135 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:24.539971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:24.539939 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv"] Nov 03 16:15:24.891387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:24.891296 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" event={"ID":"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63","Type":"ContainerStarted","Data":"930494acd9598befe0de782f7805065aec33f6c1c9be286e37104514f4a4c5b0"} Nov 03 16:15:25.001448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.001414 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:25.051371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.051333 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j"] Nov 03 16:15:25.054294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.054272 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" Nov 03 16:15:25.160183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.160080 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 16:15:25.160183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.160104 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 16:15:25.160183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.160095 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 16:15:25.160183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.160094 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rr72k\"" Nov 03 16:15:25.161406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.161381 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j"] Nov 03 16:15:25.172941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.172902 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7fs\" (UniqueName: \"kubernetes.io/projected/5a5cffea-f020-4efd-afab-ef98eecb3481-kube-api-access-vp7fs\") pod \"dns-operator-controller-manager-8557cbdb84-kdj6j\" (UID: \"5a5cffea-f020-4efd-afab-ef98eecb3481\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" Nov 03 16:15:25.273649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.273596 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7fs\" (UniqueName: \"kubernetes.io/projected/5a5cffea-f020-4efd-afab-ef98eecb3481-kube-api-access-vp7fs\") pod \"dns-operator-controller-manager-8557cbdb84-kdj6j\" (UID: \"5a5cffea-f020-4efd-afab-ef98eecb3481\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" Nov 03 16:15:25.329837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.329787 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:25.334844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.334820 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7fs\" (UniqueName: \"kubernetes.io/projected/5a5cffea-f020-4efd-afab-ef98eecb3481-kube-api-access-vp7fs\") pod \"dns-operator-controller-manager-8557cbdb84-kdj6j\" (UID: \"5a5cffea-f020-4efd-afab-ef98eecb3481\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" Nov 03 16:15:25.366680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.366650 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" Nov 03 16:15:25.549342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.549313 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:25.748178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.748024 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j"] Nov 03 16:15:25.751141 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:25.751118 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5cffea_f020_4efd_afab_ef98eecb3481.slice/crio-2da2175d5dfc85e382c7c3c927fa7143014b96ff1624e7eafab582bb90144a64 WatchSource:0}: Error finding container 2da2175d5dfc85e382c7c3c927fa7143014b96ff1624e7eafab582bb90144a64: Status 404 returned error can't find the container with id 2da2175d5dfc85e382c7c3c927fa7143014b96ff1624e7eafab582bb90144a64 Nov 03 16:15:25.819750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.819717 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw"] Nov 03 16:15:25.824763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.823490 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:25.881002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.880962 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfgw7\" (UniqueName: \"kubernetes.io/projected/7fbe9190-71bb-4143-846a-f7a4be6da47a-kube-api-access-wfgw7\") pod \"cert-manager-cainjector-5c57bbd4fc-w4dqw\" (UID: \"7fbe9190-71bb-4143-846a-f7a4be6da47a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:25.881180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.881021 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fbe9190-71bb-4143-846a-f7a4be6da47a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w4dqw\" (UID: \"7fbe9190-71bb-4143-846a-f7a4be6da47a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:25.883691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.883667 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9dpzj\"" Nov 03 16:15:25.884362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.884341 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw"] Nov 03 16:15:25.896892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.896860 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" event={"ID":"5a5cffea-f020-4efd-afab-ef98eecb3481","Type":"ContainerStarted","Data":"2da2175d5dfc85e382c7c3c927fa7143014b96ff1624e7eafab582bb90144a64"} Nov 03 16:15:25.981813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.981774 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgw7\" (UniqueName: \"kubernetes.io/projected/7fbe9190-71bb-4143-846a-f7a4be6da47a-kube-api-access-wfgw7\") pod \"cert-manager-cainjector-5c57bbd4fc-w4dqw\" (UID: \"7fbe9190-71bb-4143-846a-f7a4be6da47a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:25.981997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:25.981830 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fbe9190-71bb-4143-846a-f7a4be6da47a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w4dqw\" (UID: \"7fbe9190-71bb-4143-846a-f7a4be6da47a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:26.008702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.008622 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:26.058260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.058229 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fbe9190-71bb-4143-846a-f7a4be6da47a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w4dqw\" (UID: \"7fbe9190-71bb-4143-846a-f7a4be6da47a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:26.058449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.058426 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfgw7\" (UniqueName: \"kubernetes.io/projected/7fbe9190-71bb-4143-846a-f7a4be6da47a-kube-api-access-wfgw7\") pod \"cert-manager-cainjector-5c57bbd4fc-w4dqw\" (UID: \"7fbe9190-71bb-4143-846a-f7a4be6da47a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:26.135155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.135127 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" Nov 03 16:15:26.210932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.210887 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:26.507306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.507267 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw"] Nov 03 16:15:26.511246 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:26.511190 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fbe9190_71bb_4143_846a_f7a4be6da47a.slice/crio-501f4a0e5dd7272877340e375de541777e12c9ec6efd5d19992a4dea12560b7a WatchSource:0}: Error finding container 501f4a0e5dd7272877340e375de541777e12c9ec6efd5d19992a4dea12560b7a: Status 404 returned error can't find the container with id 501f4a0e5dd7272877340e375de541777e12c9ec6efd5d19992a4dea12560b7a Nov 03 16:15:26.773788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.773702 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:26.902121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:26.902090 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" event={"ID":"7fbe9190-71bb-4143-846a-f7a4be6da47a","Type":"ContainerStarted","Data":"501f4a0e5dd7272877340e375de541777e12c9ec6efd5d19992a4dea12560b7a"} Nov 03 16:15:27.226039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:27.225957 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:27.436049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:27.436013 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:27.632925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:27.632888 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:29.872623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:29.872562 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-d9njg" Nov 03 16:15:30.815162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:30.815114 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:30.927694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:30.927609 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" event={"ID":"b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63","Type":"ContainerStarted","Data":"e8cc3853dfdeddbda7554b9d2ebbdf7eaa69a76f2bc03c5947fe7ceccff0e83e"} Nov 03 16:15:30.928486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:30.928466 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:31.011006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:31.010969 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:31.049536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:31.049473 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" podStartSLOduration=1.9887590899999998 podStartE2EDuration="8.049452931s" totalImagesPullingTime="6.060693841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:31.048273462 +0000 UTC m=+201.169326285" watchObservedRunningTime="2026-11-03 16:15:31.049452931 +0000 UTC m=+201.170505756" Nov 03 16:15:31.934073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:31.934041 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" event={"ID":"5a5cffea-f020-4efd-afab-ef98eecb3481","Type":"ContainerStarted","Data":"692ef33fc6e6e06e451fd3e77af94f03d40e6e5c9e7db43145f0654b622563a1"} Nov 03 16:15:31.934473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:31.934259 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" Nov 03 16:15:32.034530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:32.034502 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:32.056953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:32.056903 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" podStartSLOduration=1.196687061 podStartE2EDuration="7.056886494s" totalImagesPullingTime="5.860199433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:32.055111223 +0000 UTC m=+202.176164045" watchObservedRunningTime="2026-11-03 16:15:32.056886494 +0000 UTC m=+202.177939318" Nov 03 16:15:32.940339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:32.940301 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" event={"ID":"7fbe9190-71bb-4143-846a-f7a4be6da47a","Type":"ContainerStarted","Data":"56c5062872950811a8664423a77fdca8af925a221059c50ec1010c369e615eb3"} Nov 03 16:15:33.060301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:33.060250 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w4dqw" podStartSLOduration=2.649302699 podStartE2EDuration="8.060233361s" totalImagesPullingTime="5.410930662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:33.058881454 +0000 UTC m=+203.179934282" watchObservedRunningTime="2026-11-03 16:15:33.060233361 +0000 UTC m=+203.181286182" Nov 03 16:15:33.165400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:33.165358 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:33.689726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:33.689688 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:33.921847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:33.921809 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:34.129784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:34.129748 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:35.011740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:35.011706 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:35.207147 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:35.207114 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:35.429628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:35.429546 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:35.653603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:35.653570 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:37.663426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.663392 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d"] Nov 03 16:15:37.667084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.667055 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:37.777628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.777600 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 16:15:37.777787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.777673 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-4tx7z\"" Nov 03 16:15:37.778825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.778806 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d"] Nov 03 16:15:37.790235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.790183 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d7d912-5ea6-427e-a56f-bbc2ea97b744-tmp\") pod \"jobset-operator-674cd59dc5-6bk9d\" (UID: \"58d7d912-5ea6-427e-a56f-bbc2ea97b744\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:37.790530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.790494 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pks8\" (UniqueName: \"kubernetes.io/projected/58d7d912-5ea6-427e-a56f-bbc2ea97b744-kube-api-access-5pks8\") pod \"jobset-operator-674cd59dc5-6bk9d\" (UID: \"58d7d912-5ea6-427e-a56f-bbc2ea97b744\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:37.828659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.828628 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:15:37.891656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.891619 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pks8\" (UniqueName: \"kubernetes.io/projected/58d7d912-5ea6-427e-a56f-bbc2ea97b744-kube-api-access-5pks8\") pod \"jobset-operator-674cd59dc5-6bk9d\" (UID: \"58d7d912-5ea6-427e-a56f-bbc2ea97b744\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:37.891833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.891690 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d7d912-5ea6-427e-a56f-bbc2ea97b744-tmp\") pod \"jobset-operator-674cd59dc5-6bk9d\" (UID: \"58d7d912-5ea6-427e-a56f-bbc2ea97b744\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:37.892077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.892059 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d7d912-5ea6-427e-a56f-bbc2ea97b744-tmp\") pod \"jobset-operator-674cd59dc5-6bk9d\" (UID: \"58d7d912-5ea6-427e-a56f-bbc2ea97b744\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:37.942522 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.942428 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" Nov 03 16:15:37.954482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.954458 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pks8\" (UniqueName: \"kubernetes.io/projected/58d7d912-5ea6-427e-a56f-bbc2ea97b744-kube-api-access-5pks8\") pod \"jobset-operator-674cd59dc5-6bk9d\" (UID: \"58d7d912-5ea6-427e-a56f-bbc2ea97b744\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:37.979441 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:37.979414 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" Nov 03 16:15:38.355403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:38.355354 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d"] Nov 03 16:15:38.358073 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:38.358047 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58d7d912_5ea6_427e_a56f_bbc2ea97b744.slice/crio-65e7227ad26f3e312afa31495b6dbf11ed35ed5de17013ee64656941aec00b18 WatchSource:0}: Error finding container 65e7227ad26f3e312afa31495b6dbf11ed35ed5de17013ee64656941aec00b18: Status 404 returned error can't find the container with id 65e7227ad26f3e312afa31495b6dbf11ed35ed5de17013ee64656941aec00b18 Nov 03 16:15:38.758904 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:38.758876 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:38.967042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:38.967010 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" event={"ID":"58d7d912-5ea6-427e-a56f-bbc2ea97b744","Type":"ContainerStarted","Data":"65e7227ad26f3e312afa31495b6dbf11ed35ed5de17013ee64656941aec00b18"} Nov 03 16:15:41.001571 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:41.001539 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:42.583640 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:15:42.583604 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:15:42.943454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:42.943380 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kdj6j" Nov 03 16:15:43.881841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:43.881812 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:43.905146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:43.902266 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-c8fdp"] Nov 03 16:15:43.906860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:43.906833 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:43.967636 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:43.967609 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-29djl\"" Nov 03 16:15:43.968307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:43.968285 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-c8fdp"] Nov 03 16:15:44.046237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.046136 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8fa9d43-706c-4cfa-9860-a5b2a3681a74-bound-sa-token\") pod \"cert-manager-597b95cb84-c8fdp\" (UID: \"d8fa9d43-706c-4cfa-9860-a5b2a3681a74\") " pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:44.046237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.046170 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7vr\" (UniqueName: \"kubernetes.io/projected/d8fa9d43-706c-4cfa-9860-a5b2a3681a74-kube-api-access-7x7vr\") pod \"cert-manager-597b95cb84-c8fdp\" (UID: \"d8fa9d43-706c-4cfa-9860-a5b2a3681a74\") " pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:44.118230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.118180 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:44.147359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.147322 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8fa9d43-706c-4cfa-9860-a5b2a3681a74-bound-sa-token\") pod \"cert-manager-597b95cb84-c8fdp\" (UID: \"d8fa9d43-706c-4cfa-9860-a5b2a3681a74\") " pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:44.147359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.147358 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7vr\" (UniqueName: \"kubernetes.io/projected/d8fa9d43-706c-4cfa-9860-a5b2a3681a74-kube-api-access-7x7vr\") pod \"cert-manager-597b95cb84-c8fdp\" (UID: \"d8fa9d43-706c-4cfa-9860-a5b2a3681a74\") " pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:44.208488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.208455 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7vr\" (UniqueName: \"kubernetes.io/projected/d8fa9d43-706c-4cfa-9860-a5b2a3681a74-kube-api-access-7x7vr\") pod \"cert-manager-597b95cb84-c8fdp\" (UID: \"d8fa9d43-706c-4cfa-9860-a5b2a3681a74\") " pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:44.208861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.208838 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8fa9d43-706c-4cfa-9860-a5b2a3681a74-bound-sa-token\") pod \"cert-manager-597b95cb84-c8fdp\" (UID: \"d8fa9d43-706c-4cfa-9860-a5b2a3681a74\") " pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:44.217670 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.217638 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-c8fdp" Nov 03 16:15:44.592758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.592736 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-c8fdp"] Nov 03 16:15:44.595329 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:44.595301 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8fa9d43_706c_4cfa_9860_a5b2a3681a74.slice/crio-b0040b70dc8229752d7af540b7fb8d7a4e4fe75eb0ae7019ae780215c48e8ce4 WatchSource:0}: Error finding container b0040b70dc8229752d7af540b7fb8d7a4e4fe75eb0ae7019ae780215c48e8ce4: Status 404 returned error can't find the container with id b0040b70dc8229752d7af540b7fb8d7a4e4fe75eb0ae7019ae780215c48e8ce4 Nov 03 16:15:44.997027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.996988 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-c8fdp" event={"ID":"d8fa9d43-706c-4cfa-9860-a5b2a3681a74","Type":"ContainerStarted","Data":"498c68d99dfb960487b546df65b5ebe91ad2bfa6f05c1d673cb0d0013c2c97b2"} Nov 03 16:15:44.997480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:44.997039 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-c8fdp" event={"ID":"d8fa9d43-706c-4cfa-9860-a5b2a3681a74","Type":"ContainerStarted","Data":"b0040b70dc8229752d7af540b7fb8d7a4e4fe75eb0ae7019ae780215c48e8ce4"} Nov 03 16:15:45.129590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:45.129544 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-c8fdp" podStartSLOduration=2.129529873 podStartE2EDuration="2.129529873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:45.128469201 +0000 UTC m=+215.249522025" watchObservedRunningTime="2026-11-03 16:15:45.129529873 +0000 UTC m=+215.250582695" Nov 03 16:15:45.483559 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:45.483520 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:45.684890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:45.684851 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:47.879618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:47.879573 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:47.914356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:47.914316 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2"] Nov 03 16:15:47.917079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:47.917047 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:47.980739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:47.980681 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 16:15:48.023100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.023019 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xgr22\"" Nov 03 16:15:48.023100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.023101 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 16:15:48.024604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.024577 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2"] Nov 03 16:15:48.084530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.084491 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe8b267-f4d9-4984-80ca-fca687791b68-tmp\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.084828 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.084557 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9fh\" (UniqueName: \"kubernetes.io/projected/dfe8b267-f4d9-4984-80ca-fca687791b68-kube-api-access-4w9fh\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.084828 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.084648 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe8b267-f4d9-4984-80ca-fca687791b68-tls-certs\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.171389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.171305 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:48.186234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.186170 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe8b267-f4d9-4984-80ca-fca687791b68-tmp\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.186411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.186292 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9fh\" (UniqueName: \"kubernetes.io/projected/dfe8b267-f4d9-4984-80ca-fca687791b68-kube-api-access-4w9fh\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.186411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.186345 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe8b267-f4d9-4984-80ca-fca687791b68-tls-certs\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.188433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.188410 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe8b267-f4d9-4984-80ca-fca687791b68-tmp\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.189056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.189031 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe8b267-f4d9-4984-80ca-fca687791b68-tls-certs\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.252140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.252108 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9fh\" (UniqueName: \"kubernetes.io/projected/dfe8b267-f4d9-4984-80ca-fca687791b68-kube-api-access-4w9fh\") pod \"kube-auth-proxy-5fbcdb5d6b-dzvh2\" (UID: \"dfe8b267-f4d9-4984-80ca-fca687791b68\") " pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.387581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.387552 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:48.529919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.529881 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" Nov 03 16:15:48.579049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.579017 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:48.826020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.825950 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:48.923358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:48.923250 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2"] Nov 03 16:15:48.925652 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:48.925618 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe8b267_f4d9_4984_80ca_fca687791b68.slice/crio-daeed745f57ca76d6b038801fe14b1a77b5f9f08073f7893194c8b3a4cbe9d58 WatchSource:0}: Error finding container daeed745f57ca76d6b038801fe14b1a77b5f9f08073f7893194c8b3a4cbe9d58: Status 404 returned error can't find the container with id daeed745f57ca76d6b038801fe14b1a77b5f9f08073f7893194c8b3a4cbe9d58 Nov 03 16:15:49.016427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:49.016382 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" event={"ID":"dfe8b267-f4d9-4984-80ca-fca687791b68","Type":"ContainerStarted","Data":"daeed745f57ca76d6b038801fe14b1a77b5f9f08073f7893194c8b3a4cbe9d58"} Nov 03 16:15:49.906876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:49.906838 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:50.022000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:50.021959 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" event={"ID":"58d7d912-5ea6-427e-a56f-bbc2ea97b744","Type":"ContainerStarted","Data":"03e2f4764f86f63befd4ce55c9dd0609f3d8dbe8c468a7196bffa478f77e4014"} Nov 03 16:15:50.117760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:50.117730 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:50.144986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:50.144938 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6bk9d" podStartSLOduration=2.040332555 podStartE2EDuration="13.14491835s" totalImagesPullingTime="11.104585795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:50.143051895 +0000 UTC m=+220.264104718" watchObservedRunningTime="2026-11-03 16:15:50.14491835 +0000 UTC m=+220.265971174" Nov 03 16:15:50.966868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:50.966830 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:51.194933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:51.194903 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:51.602475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:51.602440 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:51.839529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:51.839499 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:52.044215 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.044181 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:52.366835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.366755 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-tg599"] Nov 03 16:15:52.370029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.370011 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" Nov 03 16:15:52.377477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.377449 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:15:52.380898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.380876 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.426561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.426535 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zd89p\"" Nov 03 16:15:52.428075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.428049 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/638519fb-1cb1-4d77-82b6-b29e1c8981f4-config-file\") pod \"limitador-limitador-7c9b87d574-v7f8v\" (UID: \"638519fb-1cb1-4d77-82b6-b29e1c8981f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.428246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.428198 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wg75\" (UniqueName: \"kubernetes.io/projected/638519fb-1cb1-4d77-82b6-b29e1c8981f4-kube-api-access-6wg75\") pod \"limitador-limitador-7c9b87d574-v7f8v\" (UID: \"638519fb-1cb1-4d77-82b6-b29e1c8981f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.428304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.428278 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvh6\" (UniqueName: \"kubernetes.io/projected/caac17b6-2bd9-4903-ba9c-7fbbabac6870-kube-api-access-hcvh6\") pod \"authorino-56c6c9ddd5-tg599\" (UID: \"caac17b6-2bd9-4903-ba9c-7fbbabac6870\") " pod="kuadrant-system/authorino-56c6c9ddd5-tg599" Nov 03 16:15:52.432558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.432528 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-tg599"] Nov 03 16:15:52.438130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.438108 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 16:15:52.478117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.478088 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:15:52.527083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.527053 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jjqwv\"" Nov 03 16:15:52.529473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.529450 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wg75\" (UniqueName: \"kubernetes.io/projected/638519fb-1cb1-4d77-82b6-b29e1c8981f4-kube-api-access-6wg75\") pod \"limitador-limitador-7c9b87d574-v7f8v\" (UID: \"638519fb-1cb1-4d77-82b6-b29e1c8981f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.529548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.529490 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvh6\" (UniqueName: \"kubernetes.io/projected/caac17b6-2bd9-4903-ba9c-7fbbabac6870-kube-api-access-hcvh6\") pod \"authorino-56c6c9ddd5-tg599\" (UID: \"caac17b6-2bd9-4903-ba9c-7fbbabac6870\") " pod="kuadrant-system/authorino-56c6c9ddd5-tg599" Nov 03 16:15:52.529548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.529513 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/638519fb-1cb1-4d77-82b6-b29e1c8981f4-config-file\") pod \"limitador-limitador-7c9b87d574-v7f8v\" (UID: \"638519fb-1cb1-4d77-82b6-b29e1c8981f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.530079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.530063 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/638519fb-1cb1-4d77-82b6-b29e1c8981f4-config-file\") pod \"limitador-limitador-7c9b87d574-v7f8v\" (UID: \"638519fb-1cb1-4d77-82b6-b29e1c8981f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.569315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.569286 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:52.602820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.602791 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wg75\" (UniqueName: \"kubernetes.io/projected/638519fb-1cb1-4d77-82b6-b29e1c8981f4-kube-api-access-6wg75\") pod \"limitador-limitador-7c9b87d574-v7f8v\" (UID: \"638519fb-1cb1-4d77-82b6-b29e1c8981f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.604910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.604881 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvh6\" (UniqueName: \"kubernetes.io/projected/caac17b6-2bd9-4903-ba9c-7fbbabac6870-kube-api-access-hcvh6\") pod \"authorino-56c6c9ddd5-tg599\" (UID: \"caac17b6-2bd9-4903-ba9c-7fbbabac6870\") " pod="kuadrant-system/authorino-56c6c9ddd5-tg599" Nov 03 16:15:52.684072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.684003 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" Nov 03 16:15:52.693345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.693323 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:52.696397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.696374 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:15:52.768612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:52.767856 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:53.064049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:53.064012 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-tg599"] Nov 03 16:15:53.070159 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:53.070128 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaac17b6_2bd9_4903_ba9c_7fbbabac6870.slice/crio-2994ae777980b15661614c8ce03a758fb884d38e245bbcf726fe195d94e7af00 WatchSource:0}: Error finding container 2994ae777980b15661614c8ce03a758fb884d38e245bbcf726fe195d94e7af00: Status 404 returned error can't find the container with id 2994ae777980b15661614c8ce03a758fb884d38e245bbcf726fe195d94e7af00 Nov 03 16:15:53.116010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:53.115987 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:15:53.117969 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:15:53.117944 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638519fb_1cb1_4d77_82b6_b29e1c8981f4.slice/crio-d4bcd7b15bb194032b163d79ea33157931ef829d8b41dc9fc7b7b374c30dd8a2 WatchSource:0}: Error finding container d4bcd7b15bb194032b163d79ea33157931ef829d8b41dc9fc7b7b374c30dd8a2: Status 404 returned error can't find the container with id d4bcd7b15bb194032b163d79ea33157931ef829d8b41dc9fc7b7b374c30dd8a2 Nov 03 16:15:53.983906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:53.983874 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:54.043080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:54.043048 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" event={"ID":"caac17b6-2bd9-4903-ba9c-7fbbabac6870","Type":"ContainerStarted","Data":"2994ae777980b15661614c8ce03a758fb884d38e245bbcf726fe195d94e7af00"} Nov 03 16:15:54.044495 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:54.044468 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" event={"ID":"dfe8b267-f4d9-4984-80ca-fca687791b68","Type":"ContainerStarted","Data":"15ebdda32735607b0c6d349fe796e3db552266b924d6cf2db0e148d659e63722"} Nov 03 16:15:54.045585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:54.045564 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" event={"ID":"638519fb-1cb1-4d77-82b6-b29e1c8981f4","Type":"ContainerStarted","Data":"d4bcd7b15bb194032b163d79ea33157931ef829d8b41dc9fc7b7b374c30dd8a2"} Nov 03 16:15:54.172189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:54.172136 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fbcdb5d6b-dzvh2" podStartSLOduration=2.458439651 podStartE2EDuration="7.17212402s" totalImagesPullingTime="4.713684369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:54.171403777 +0000 UTC m=+224.292456599" watchObservedRunningTime="2026-11-03 16:15:54.17212402 +0000 UTC m=+224.293176843" Nov 03 16:15:55.315047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:55.314979 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:56.057469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:56.057435 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" event={"ID":"638519fb-1cb1-4d77-82b6-b29e1c8981f4","Type":"ContainerStarted","Data":"9be630346a8665e8f6f3a0d1ba19e08f2a1b7875b96024ea2e8bf7603dcd3147"} Nov 03 16:15:56.057665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:56.057573 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:15:56.191615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:56.191549 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" podStartSLOduration=1.984332008 podStartE2EDuration="4.191529615s" totalImagesPullingTime="2.207197607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:56.190518868 +0000 UTC m=+226.311571690" watchObservedRunningTime="2026-11-03 16:15:56.191529615 +0000 UTC m=+226.312582442" Nov 03 16:15:57.063320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:57.063281 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" event={"ID":"caac17b6-2bd9-4903-ba9c-7fbbabac6870","Type":"ContainerStarted","Data":"537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930"} Nov 03 16:15:57.181817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:57.181766 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" podStartSLOduration=1.7410785199999999 podStartE2EDuration="5.181752438s" totalImagesPullingTime="3.440673918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:57.181685072 +0000 UTC m=+227.302737898" watchObservedRunningTime="2026-11-03 16:15:57.181752438 +0000 UTC m=+227.302805260" Nov 03 16:15:59.833441 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:15:59.833390 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:00.033071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:00.033036 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:00.240240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:00.240186 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:02.438744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:02.438702 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:02.633688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:02.633657 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:02.837058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:02.837015 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:05.399022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:05.398978 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:05.590890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:05.590848 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:07.064440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:07.064409 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v7f8v" Nov 03 16:16:10.089476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:10.089446 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:10.294017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:10.293973 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:12.584339 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:16:12.584301 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:16:18.918340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:18.918298 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:19.173584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:19.173452 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:20.415708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:20.415667 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:20.627903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:20.627870 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:21.904086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:21.904040 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:22.107581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:22.107536 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:23.282678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:23.282644 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:23.481218 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:23.481173 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:24.523609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:24.523577 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:24.714902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:24.714862 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:25.352830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:25.352790 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:29.920358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:29.920321 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:30.126878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:30.126828 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:30.333449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:30.333405 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:30.517805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:30.517773 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:30.710806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:30.710728 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:30.912556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:30.912524 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:33.834019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:33.833981 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:34.024609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:34.024571 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:34.676738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:34.676708 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:38.731214 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:38.731165 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:38.934969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:38.934931 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:41.720188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:41.720157 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:41.930709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:41.930667 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:42.584764 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:16:42.584730 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:16:57.680672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.680638 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:57.717028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.716998 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-52vwg"] Nov 03 16:16:57.721580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.721564 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.777036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.777014 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 16:16:57.783699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.783677 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 16:16:57.816185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.816159 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqntb\" (UniqueName: \"kubernetes.io/projected/2ac43f53-c06d-4c28-b591-864fc18f43ee-kube-api-access-gqntb\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.816305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.816197 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-metrics-certs\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.816365 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.816316 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-webhook-cert\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.827101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.827074 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 16:16:57.827101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.827089 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 16:16:57.827101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.827096 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rfldq\"" Nov 03 16:16:57.827317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.827246 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 16:16:57.829433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.829406 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-52vwg"] Nov 03 16:16:57.875683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.875641 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:57.916801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.916764 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqntb\" (UniqueName: \"kubernetes.io/projected/2ac43f53-c06d-4c28-b591-864fc18f43ee-kube-api-access-gqntb\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.916801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.916804 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-metrics-certs\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.917001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.916854 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-webhook-cert\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.919465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.919434 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-webhook-cert\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.919592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.919495 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-metrics-certs\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:57.977647 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:57.977578 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqntb\" (UniqueName: \"kubernetes.io/projected/2ac43f53-c06d-4c28-b591-864fc18f43ee-kube-api-access-gqntb\") pod \"maas-controller-579cf7c6d-52vwg\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:58.034412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.034371 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:16:58.084852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.084819 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:58.159308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.159275 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b46f6b789-w5sxs"] Nov 03 16:16:58.163041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.163018 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.219635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.219600 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-metrics-certs\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.219826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.219660 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-webhook-cert\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.219826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.219750 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkkn\" (UniqueName: \"kubernetes.io/projected/46d86ba8-5de5-42ed-a4b4-7201dc969002-kube-api-access-vrkkn\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.224165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.224137 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-w5sxs"] Nov 03 16:16:58.289712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.289682 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:58.320694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.320652 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-webhook-cert\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.320911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.320729 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkkn\" (UniqueName: \"kubernetes.io/projected/46d86ba8-5de5-42ed-a4b4-7201dc969002-kube-api-access-vrkkn\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.320911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.320811 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-metrics-certs\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.324255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.324228 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-webhook-cert\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.324401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.324318 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-metrics-certs\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.382182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.382154 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkkn\" (UniqueName: \"kubernetes.io/projected/46d86ba8-5de5-42ed-a4b4-7201dc969002-kube-api-access-vrkkn\") pod \"maas-controller-6b46f6b789-w5sxs\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.417905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.417877 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-52vwg"] Nov 03 16:16:58.420556 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:16:58.420528 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac43f53_c06d_4c28_b591_864fc18f43ee.slice/crio-dd65eeaaf3ea23134a5829dc53ac244b8547fd0a490f1230df04210ae26134ea WatchSource:0}: Error finding container dd65eeaaf3ea23134a5829dc53ac244b8547fd0a490f1230df04210ae26134ea: Status 404 returned error can't find the container with id dd65eeaaf3ea23134a5829dc53ac244b8547fd0a490f1230df04210ae26134ea Nov 03 16:16:58.477045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.477001 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:16:58.852738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:58.852705 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-w5sxs"] Nov 03 16:16:58.855510 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:16:58.855483 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d86ba8_5de5_42ed_a4b4_7201dc969002.slice/crio-4c4c6965a003e69e51cfcf097767a95c7192496dac2f94d8a20fba094918b92a WatchSource:0}: Error finding container 4c4c6965a003e69e51cfcf097767a95c7192496dac2f94d8a20fba094918b92a: Status 404 returned error can't find the container with id 4c4c6965a003e69e51cfcf097767a95c7192496dac2f94d8a20fba094918b92a Nov 03 16:16:59.356754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:59.356718 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-52vwg" event={"ID":"2ac43f53-c06d-4c28-b591-864fc18f43ee","Type":"ContainerStarted","Data":"dd65eeaaf3ea23134a5829dc53ac244b8547fd0a490f1230df04210ae26134ea"} Nov 03 16:16:59.357885 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:16:59.357863 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" event={"ID":"46d86ba8-5de5-42ed-a4b4-7201dc969002","Type":"ContainerStarted","Data":"4c4c6965a003e69e51cfcf097767a95c7192496dac2f94d8a20fba094918b92a"} Nov 03 16:17:04.387913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.387817 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-52vwg" event={"ID":"2ac43f53-c06d-4c28-b591-864fc18f43ee","Type":"ContainerStarted","Data":"4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d"} Nov 03 16:17:04.387913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.387885 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:17:04.389490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.389464 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" event={"ID":"46d86ba8-5de5-42ed-a4b4-7201dc969002","Type":"ContainerStarted","Data":"786cd5a3590f881db6c2e91641bd39b9e650666f0671c6b93d0de5c34cf44427"} Nov 03 16:17:04.389613 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.389525 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:17:04.448101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.448070 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:04.514135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.514083 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-52vwg" podStartSLOduration=1.89152732 podStartE2EDuration="7.514067807s" totalImagesPullingTime="5.622540487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:04.512831153 +0000 UTC m=+294.633883975" watchObservedRunningTime="2026-11-03 16:17:04.514067807 +0000 UTC m=+294.635120630" Nov 03 16:17:04.635531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.635474 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" podStartSLOduration=1.488415656 podStartE2EDuration="6.635458092s" totalImagesPullingTime="5.147042436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:04.634146745 +0000 UTC m=+294.755199568" watchObservedRunningTime="2026-11-03 16:17:04.635458092 +0000 UTC m=+294.756510914" Nov 03 16:17:04.643940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:04.643865 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:07.415173 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.415137 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf"] Nov 03 16:17:07.418595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.418578 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.474174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.474125 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 16:17:07.474174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.474154 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 16:17:07.523251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.518258 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94f8l\" (UniqueName: \"kubernetes.io/projected/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-kube-api-access-94f8l\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.523251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.518351 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.523251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.518394 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.523251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.518466 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.526070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.526002 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 16:17:07.526251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.526224 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-4266c\"" Nov 03 16:17:07.527785 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.527753 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf"] Nov 03 16:17:07.606599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.606565 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:07.619735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.619703 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.619888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.619742 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94f8l\" (UniqueName: \"kubernetes.io/projected/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-kube-api-access-94f8l\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.619888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.619784 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.619888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.619810 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.619888 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:07.619870 5946 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 16:17:07.620109 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:07.619921 5946 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 16:17:07.620109 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:07.619958 5946 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-proxy-source podName:c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd nodeName:}" failed. No retries permitted until 2026-11-03 16:17:08.11993176 +0000 UTC m=+298.240984578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" (UID: "c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd") : configmap "usage-tenancy-proxy-source" not found Nov 03 16:17:07.620109 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:07.619979 5946 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-tls-cert podName:c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd nodeName:}" failed. No retries permitted until 2026-11-03 16:17:08.119969341 +0000 UTC m=+298.241022152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" (UID: "c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 16:17:07.620605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.620588 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.685151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.685077 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94f8l\" (UniqueName: \"kubernetes.io/projected/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-kube-api-access-94f8l\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:07.805361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:07.805327 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:08.125170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:08.125114 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:08.125402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:08.125195 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:08.126043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:08.126022 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:08.127816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:08.127790 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p2gwf\" (UID: \"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:08.331093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:08.331058 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:08.710882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:08.710859 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf"] Nov 03 16:17:08.712720 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:17:08.712687 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e8b5ac_ddb5_42ed_a5f6_0e2b266441fd.slice/crio-be81ef80f2e2b2dce84bd80ca1b1b3fd7624cd8c0c5fa071ce106ea496d6f1c0 WatchSource:0}: Error finding container be81ef80f2e2b2dce84bd80ca1b1b3fd7624cd8c0c5fa071ce106ea496d6f1c0: Status 404 returned error can't find the container with id be81ef80f2e2b2dce84bd80ca1b1b3fd7624cd8c0c5fa071ce106ea496d6f1c0 Nov 03 16:17:09.415850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:09.415812 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" event={"ID":"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd","Type":"ContainerStarted","Data":"be81ef80f2e2b2dce84bd80ca1b1b3fd7624cd8c0c5fa071ce106ea496d6f1c0"} Nov 03 16:17:10.034247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:10.034196 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:17:10.034671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:10.034470 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:17:12.118131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:12.118097 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:12.611703 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:12.611616 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:17:13.056564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:13.056495 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:15.396978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:15.396887 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 16:17:15.397968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:15.397433 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:17:15.572420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:15.572385 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-52vwg"] Nov 03 16:17:15.572716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:15.572682 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-52vwg" podUID="2ac43f53-c06d-4c28-b591-864fc18f43ee" containerName="manager" containerID="cri-o://4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d" gracePeriod=30 Nov 03 16:17:15.831745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:15.831717 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:17:16.009167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.009136 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-metrics-certs\") pod \"2ac43f53-c06d-4c28-b591-864fc18f43ee\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " Nov 03 16:17:16.009367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.009189 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-webhook-cert\") pod \"2ac43f53-c06d-4c28-b591-864fc18f43ee\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " Nov 03 16:17:16.009367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.009323 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac43f53-c06d-4c28-b591-864fc18f43ee-kube-api-access-gqntb\" (UniqueName: \"kubernetes.io/projected/2ac43f53-c06d-4c28-b591-864fc18f43ee-kube-api-access-gqntb\") pod \"2ac43f53-c06d-4c28-b591-864fc18f43ee\" (UID: \"2ac43f53-c06d-4c28-b591-864fc18f43ee\") " Nov 03 16:17:16.013404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.013354 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac43f53-c06d-4c28-b591-864fc18f43ee-kube-api-access-gqntb" pod "2ac43f53-c06d-4c28-b591-864fc18f43ee" (UID: "2ac43f53-c06d-4c28-b591-864fc18f43ee"). InnerVolumeSpecName "kube-api-access-gqntb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:17:16.018482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.018412 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-metrics-certs" pod "2ac43f53-c06d-4c28-b591-864fc18f43ee" (UID: "2ac43f53-c06d-4c28-b591-864fc18f43ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:17:16.023939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.023909 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-webhook-cert" pod "2ac43f53-c06d-4c28-b591-864fc18f43ee" (UID: "2ac43f53-c06d-4c28-b591-864fc18f43ee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:17:16.110941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.110629 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqntb\" (UniqueName: \"kubernetes.io/projected/2ac43f53-c06d-4c28-b591-864fc18f43ee-kube-api-access-gqntb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:17:16.110941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.110664 5946 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-metrics-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:17:16.110941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.110679 5946 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ac43f53-c06d-4c28-b591-864fc18f43ee-webhook-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:17:16.458586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.458503 5946 generic.go:356] "Generic (PLEG): container finished" podID="2ac43f53-c06d-4c28-b591-864fc18f43ee" containerID="4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d" exitCode=0 Nov 03 16:17:16.458586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.458555 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-52vwg" Nov 03 16:17:16.459047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.458587 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-52vwg" event={"ID":"2ac43f53-c06d-4c28-b591-864fc18f43ee","Type":"ContainerDied","Data":"4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d"} Nov 03 16:17:16.459047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.458644 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-52vwg" event={"ID":"2ac43f53-c06d-4c28-b591-864fc18f43ee","Type":"ContainerDied","Data":"dd65eeaaf3ea23134a5829dc53ac244b8547fd0a490f1230df04210ae26134ea"} Nov 03 16:17:16.459047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.458672 5946 scope.go:122] "RemoveContainer" containerID="4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d" Nov 03 16:17:16.467728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.467706 5946 scope.go:122] "RemoveContainer" containerID="4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d" Nov 03 16:17:16.467999 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:16.467975 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d\": container with ID starting with 4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d not found: ID does not exist" containerID="4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d" Nov 03 16:17:16.468044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.468015 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d"} err="failed to get container status \"4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d\": rpc error: code = NotFound desc = could not find container \"4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d\": container with ID starting with 4aebe82c532763748fc879eb293e3d7e61ff709540a71f073a3024f593ea4f6d not found: ID does not exist" Nov 03 16:17:16.694036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.693999 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-52vwg"] Nov 03 16:17:16.699362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:16.699328 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-52vwg"] Nov 03 16:17:18.003644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:18.003607 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac43f53-c06d-4c28-b591-864fc18f43ee" path="/var/lib/kubelet/pods/2ac43f53-c06d-4c28-b591-864fc18f43ee/volumes" Nov 03 16:17:21.842324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:21.842265 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:22.036909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.036871 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:22.235113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.235075 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:22.423928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.423898 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:22.644755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.644719 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:22.758719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.758688 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d"] Nov 03 16:17:22.759253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.759200 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2ac43f53-c06d-4c28-b591-864fc18f43ee" containerName="manager" Nov 03 16:17:22.759298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.759257 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac43f53-c06d-4c28-b591-864fc18f43ee" containerName="manager" Nov 03 16:17:22.759428 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.759417 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac43f53-c06d-4c28-b591-864fc18f43ee" containerName="manager" Nov 03 16:17:22.762870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.762855 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:22.770585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.770562 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eb79f6e-18f4-4fbe-9069-599dce82a523-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-kmq6d\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:22.770716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.770622 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wwf\" (UniqueName: \"kubernetes.io/projected/1eb79f6e-18f4-4fbe-9069-599dce82a523-kube-api-access-99wwf\") pod \"llmisvc-controller-manager-869d5dcc5d-kmq6d\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:22.821886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.821811 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d"] Nov 03 16:17:22.842185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.842160 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:22.870567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.870532 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2m5fn\"" Nov 03 16:17:22.870567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.870550 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 16:17:22.871985 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.871955 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eb79f6e-18f4-4fbe-9069-599dce82a523-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-kmq6d\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:22.872106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.872006 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wwf\" (UniqueName: \"kubernetes.io/projected/1eb79f6e-18f4-4fbe-9069-599dce82a523-kube-api-access-99wwf\") pod \"llmisvc-controller-manager-869d5dcc5d-kmq6d\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:22.872369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.872350 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xz8cb"] Nov 03 16:17:22.874603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.874581 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eb79f6e-18f4-4fbe-9069-599dce82a523-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-kmq6d\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:22.880593 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.880574 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:22.916837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.916810 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xz8cb"] Nov 03 16:17:22.934786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.934760 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wwf\" (UniqueName: \"kubernetes.io/projected/1eb79f6e-18f4-4fbe-9069-599dce82a523-kube-api-access-99wwf\") pod \"llmisvc-controller-manager-869d5dcc5d-kmq6d\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:22.939157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.939132 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-gf4xd\"" Nov 03 16:17:22.939281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.939166 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 16:17:22.972814 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.972780 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4208b06-4937-4c72-bec3-b23d6c6dde43-tls-certs\") pod \"model-serving-api-75bd89fdfc-xz8cb\" (UID: \"b4208b06-4937-4c72-bec3-b23d6c6dde43\") " pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:22.972939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:22.972820 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4c8\" (UniqueName: \"kubernetes.io/projected/b4208b06-4937-4c72-bec3-b23d6c6dde43-kube-api-access-fk4c8\") pod \"model-serving-api-75bd89fdfc-xz8cb\" (UID: \"b4208b06-4937-4c72-bec3-b23d6c6dde43\") " pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:23.032906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.032864 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:23.074255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.074152 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4208b06-4937-4c72-bec3-b23d6c6dde43-tls-certs\") pod \"model-serving-api-75bd89fdfc-xz8cb\" (UID: \"b4208b06-4937-4c72-bec3-b23d6c6dde43\") " pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:23.074255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.074197 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4c8\" (UniqueName: \"kubernetes.io/projected/b4208b06-4937-4c72-bec3-b23d6c6dde43-kube-api-access-fk4c8\") pod \"model-serving-api-75bd89fdfc-xz8cb\" (UID: \"b4208b06-4937-4c72-bec3-b23d6c6dde43\") " pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:23.077084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.077061 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4208b06-4937-4c72-bec3-b23d6c6dde43-tls-certs\") pod \"model-serving-api-75bd89fdfc-xz8cb\" (UID: \"b4208b06-4937-4c72-bec3-b23d6c6dde43\") " pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:23.077876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.077858 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:23.135050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.135016 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4c8\" (UniqueName: \"kubernetes.io/projected/b4208b06-4937-4c72-bec3-b23d6c6dde43-kube-api-access-fk4c8\") pod \"model-serving-api-75bd89fdfc-xz8cb\" (UID: \"b4208b06-4937-4c72-bec3-b23d6c6dde43\") " pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:23.194351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.194315 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:23.452404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.452375 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d"] Nov 03 16:17:23.455148 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:17:23.455120 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1eb79f6e_18f4_4fbe_9069_599dce82a523.slice/crio-c56b375d51a6d85b1ce6a28828a938fdcdcbc3bbb42000ea4d0e67f3023e26ab WatchSource:0}: Error finding container c56b375d51a6d85b1ce6a28828a938fdcdcbc3bbb42000ea4d0e67f3023e26ab: Status 404 returned error can't find the container with id c56b375d51a6d85b1ce6a28828a938fdcdcbc3bbb42000ea4d0e67f3023e26ab Nov 03 16:17:23.498323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.498282 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" event={"ID":"c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd","Type":"ContainerStarted","Data":"f50a13816f1c202735c3d4e519dd794df77758a3524a5f2be64f8bbd8ba0f273"} Nov 03 16:17:23.498518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.498394 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:23.499492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.499466 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" event={"ID":"1eb79f6e-18f4-4fbe-9069-599dce82a523","Type":"ContainerStarted","Data":"c56b375d51a6d85b1ce6a28828a938fdcdcbc3bbb42000ea4d0e67f3023e26ab"} Nov 03 16:17:23.570070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.570044 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xz8cb"] Nov 03 16:17:23.572748 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:17:23.572719 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4208b06_4937_4c72_bec3_b23d6c6dde43.slice/crio-a3927ec8606de5e35adcfd453732f139d4d175a8c2f5e437201642d33421e0ea WatchSource:0}: Error finding container a3927ec8606de5e35adcfd453732f139d4d175a8c2f5e437201642d33421e0ea: Status 404 returned error can't find the container with id a3927ec8606de5e35adcfd453732f139d4d175a8c2f5e437201642d33421e0ea Nov 03 16:17:23.621315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:23.621265 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" podStartSLOduration=2.801995968 podStartE2EDuration="16.621249202s" totalImagesPullingTime="13.819253234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:23.620311393 +0000 UTC m=+313.741364218" watchObservedRunningTime="2026-11-03 16:17:23.621249202 +0000 UTC m=+313.742302075" Nov 03 16:17:24.006100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:24.006068 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:24.505161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:24.505121 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" event={"ID":"b4208b06-4937-4c72-bec3-b23d6c6dde43","Type":"ContainerStarted","Data":"a3927ec8606de5e35adcfd453732f139d4d175a8c2f5e437201642d33421e0ea"} Nov 03 16:17:25.694543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:25.694470 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:26.516688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:26.516657 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" event={"ID":"b4208b06-4937-4c72-bec3-b23d6c6dde43","Type":"ContainerStarted","Data":"3d788e161ccb375c3d43c1fcf844ae65b3775df44882f9b8ffb53e23566c808d"} Nov 03 16:17:26.516876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:26.516768 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:26.518350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:26.518327 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" event={"ID":"1eb79f6e-18f4-4fbe-9069-599dce82a523","Type":"ContainerStarted","Data":"6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757"} Nov 03 16:17:26.518448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:26.518387 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:26.640934 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:26.640880 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" podStartSLOduration=2.293903568 podStartE2EDuration="4.640865516s" totalImagesPullingTime="2.346961948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:26.640513384 +0000 UTC m=+316.761566210" watchObservedRunningTime="2026-11-03 16:17:26.640865516 +0000 UTC m=+316.761918339" Nov 03 16:17:26.760455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:26.760403 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" podStartSLOduration=2.208005305 podStartE2EDuration="4.760389396s" totalImagesPullingTime="2.552384091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:26.760350332 +0000 UTC m=+316.881403156" watchObservedRunningTime="2026-11-03 16:17:26.760389396 +0000 UTC m=+316.881442248" Nov 03 16:17:28.926988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:28.926957 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:29.127644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:29.127615 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:29.325065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:29.325034 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:29.526341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:29.526313 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:29.725181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:29.725105 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:29.968986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:29.968953 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:30.170127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:30.170098 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:30.362916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:30.362883 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:34.511374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:34.511344 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p2gwf" Nov 03 16:17:37.527755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:37.527723 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-xz8cb" Nov 03 16:17:37.674996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:37.674964 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:37.865719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:37.865640 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:39.739611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:39.739564 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:39.933070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:39.933034 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:40.118644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:40.118575 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:42.584918 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:42.584877 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:17:46.194767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:46.194730 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:17:52.322598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:52.322561 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:52.511505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:52.511474 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:52.623091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:52.623003 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jkr5x"] Nov 03 16:17:52.795301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:52.795261 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jkr5x"] Nov 03 16:17:52.795465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:52.795368 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" Nov 03 16:17:52.941502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:52.941399 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfbkd\" (UniqueName: \"kubernetes.io/projected/bb0ce2a6-b585-4a66-809e-cd3f183134b6-kube-api-access-sfbkd\") pod \"authorino-6f6944bcdd-jkr5x\" (UID: \"bb0ce2a6-b585-4a66-809e-cd3f183134b6\") " pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" Nov 03 16:17:52.977046 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:52.977008 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jkr5x"] Nov 03 16:17:52.977308 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:52.977285 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sfbkd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" podUID="bb0ce2a6-b585-4a66-809e-cd3f183134b6" Nov 03 16:17:53.042942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.042913 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfbkd\" (UniqueName: \"kubernetes.io/projected/bb0ce2a6-b585-4a66-809e-cd3f183134b6-kube-api-access-sfbkd\") pod \"authorino-6f6944bcdd-jkr5x\" (UID: \"bb0ce2a6-b585-4a66-809e-cd3f183134b6\") " pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" Nov 03 16:17:53.103243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.103190 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfbkd\" (UniqueName: \"kubernetes.io/projected/bb0ce2a6-b585-4a66-809e-cd3f183134b6-kube-api-access-sfbkd\") pod \"authorino-6f6944bcdd-jkr5x\" (UID: \"bb0ce2a6-b585-4a66-809e-cd3f183134b6\") " pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" Nov 03 16:17:53.473396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.473360 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bd98557cb-dzhcq"] Nov 03 16:17:53.503404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.503374 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.535909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.535882 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd98557cb-dzhcq"] Nov 03 16:17:53.558961 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.558937 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 16:17:53.642644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.642614 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" Nov 03 16:17:53.648937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.648912 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261c7408-c4e6-45df-8a13-ed9f9558e729-tls-cert\") pod \"authorino-bd98557cb-dzhcq\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.649110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.648944 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvcx\" (UniqueName: \"kubernetes.io/projected/261c7408-c4e6-45df-8a13-ed9f9558e729-kube-api-access-4tvcx\") pod \"authorino-bd98557cb-dzhcq\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.665292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.665266 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" Nov 03 16:17:53.749583 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.749549 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0ce2a6-b585-4a66-809e-cd3f183134b6-kube-api-access-sfbkd\" (UniqueName: \"kubernetes.io/projected/bb0ce2a6-b585-4a66-809e-cd3f183134b6-kube-api-access-sfbkd\") pod \"bb0ce2a6-b585-4a66-809e-cd3f183134b6\" (UID: \"bb0ce2a6-b585-4a66-809e-cd3f183134b6\") " Nov 03 16:17:53.749772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.749699 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261c7408-c4e6-45df-8a13-ed9f9558e729-tls-cert\") pod \"authorino-bd98557cb-dzhcq\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.749772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.749736 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvcx\" (UniqueName: \"kubernetes.io/projected/261c7408-c4e6-45df-8a13-ed9f9558e729-kube-api-access-4tvcx\") pod \"authorino-bd98557cb-dzhcq\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.751926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.751893 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0ce2a6-b585-4a66-809e-cd3f183134b6-kube-api-access-sfbkd" pod "bb0ce2a6-b585-4a66-809e-cd3f183134b6" (UID: "bb0ce2a6-b585-4a66-809e-cd3f183134b6"). InnerVolumeSpecName "kube-api-access-sfbkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:17:53.752279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.752263 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261c7408-c4e6-45df-8a13-ed9f9558e729-tls-cert\") pod \"authorino-bd98557cb-dzhcq\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.809534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.809502 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvcx\" (UniqueName: \"kubernetes.io/projected/261c7408-c4e6-45df-8a13-ed9f9558e729-kube-api-access-4tvcx\") pod \"authorino-bd98557cb-dzhcq\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.827891 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.827858 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:17:53.851026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:53.850997 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfbkd\" (UniqueName: \"kubernetes.io/projected/bb0ce2a6-b585-4a66-809e-cd3f183134b6-kube-api-access-sfbkd\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:17:54.220482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:54.220451 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd98557cb-dzhcq"] Nov 03 16:17:54.223658 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:17:54.223632 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261c7408_c4e6_45df_8a13_ed9f9558e729.slice/crio-9912059d5d57a4f62f770917e863bcb844503ca32ed3b99f501f08c55f3533d9 WatchSource:0}: Error finding container 9912059d5d57a4f62f770917e863bcb844503ca32ed3b99f501f08c55f3533d9: Status 404 returned error can't find the container with id 9912059d5d57a4f62f770917e863bcb844503ca32ed3b99f501f08c55f3533d9 Nov 03 16:17:54.648694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:54.648666 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jkr5x" Nov 03 16:17:54.649172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:54.648667 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd98557cb-dzhcq" event={"ID":"261c7408-c4e6-45df-8a13-ed9f9558e729","Type":"ContainerStarted","Data":"9912059d5d57a4f62f770917e863bcb844503ca32ed3b99f501f08c55f3533d9"} Nov 03 16:17:54.882086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:54.882046 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jkr5x"] Nov 03 16:17:54.890712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:54.890690 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jkr5x"] Nov 03 16:17:56.003049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:56.002949 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0ce2a6-b585-4a66-809e-cd3f183134b6" path="/var/lib/kubelet/pods/bb0ce2a6-b585-4a66-809e-cd3f183134b6/volumes" Nov 03 16:17:56.667076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:56.666978 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd98557cb-dzhcq" event={"ID":"261c7408-c4e6-45df-8a13-ed9f9558e729","Type":"ContainerStarted","Data":"5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b"} Nov 03 16:17:56.786678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:56.786618 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-bd98557cb-dzhcq" podStartSLOduration=1.787749801 podStartE2EDuration="3.786604163s" totalImagesPullingTime="1.998854362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:56.785635324 +0000 UTC m=+346.906688141" watchObservedRunningTime="2026-11-03 16:17:56.786604163 +0000 UTC m=+346.907656986" Nov 03 16:17:56.838144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:56.838112 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-tg599"] Nov 03 16:17:56.838364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:56.838343 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" podUID="caac17b6-2bd9-4903-ba9c-7fbbabac6870" containerName="authorino" containerID="cri-o://537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930" gracePeriod=30 Nov 03 16:17:57.127399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.127371 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" Nov 03 16:17:57.283267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.283196 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caac17b6-2bd9-4903-ba9c-7fbbabac6870-kube-api-access-hcvh6\" (UniqueName: \"kubernetes.io/projected/caac17b6-2bd9-4903-ba9c-7fbbabac6870-kube-api-access-hcvh6\") pod \"caac17b6-2bd9-4903-ba9c-7fbbabac6870\" (UID: \"caac17b6-2bd9-4903-ba9c-7fbbabac6870\") " Nov 03 16:17:57.285558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.285523 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caac17b6-2bd9-4903-ba9c-7fbbabac6870-kube-api-access-hcvh6" pod "caac17b6-2bd9-4903-ba9c-7fbbabac6870" (UID: "caac17b6-2bd9-4903-ba9c-7fbbabac6870"). InnerVolumeSpecName "kube-api-access-hcvh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:17:57.384778 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.384747 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvh6\" (UniqueName: \"kubernetes.io/projected/caac17b6-2bd9-4903-ba9c-7fbbabac6870-kube-api-access-hcvh6\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:17:57.525027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.524997 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:17:57.673644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.673550 5946 generic.go:356] "Generic (PLEG): container finished" podID="caac17b6-2bd9-4903-ba9c-7fbbabac6870" containerID="537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930" exitCode=0 Nov 03 16:17:57.673644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.673605 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" Nov 03 16:17:57.673844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.673631 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" event={"ID":"caac17b6-2bd9-4903-ba9c-7fbbabac6870","Type":"ContainerDied","Data":"537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930"} Nov 03 16:17:57.673844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.673678 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-tg599" event={"ID":"caac17b6-2bd9-4903-ba9c-7fbbabac6870","Type":"ContainerDied","Data":"2994ae777980b15661614c8ce03a758fb884d38e245bbcf726fe195d94e7af00"} Nov 03 16:17:57.673844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.673700 5946 scope.go:122] "RemoveContainer" containerID="537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930" Nov 03 16:17:57.684554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.684533 5946 scope.go:122] "RemoveContainer" containerID="537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930" Nov 03 16:17:57.684896 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:17:57.684864 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930\": container with ID starting with 537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930 not found: ID does not exist" containerID="537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930" Nov 03 16:17:57.684981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.684927 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930"} err="failed to get container status \"537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930\": rpc error: code = NotFound desc = could not find container \"537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930\": container with ID starting with 537cfc9d57e0948d90b061ca76cde892d29d3caa94b069a9055997885bdb7930 not found: ID does not exist" Nov 03 16:17:57.845070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.845033 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-tg599"] Nov 03 16:17:57.848507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:57.848467 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-tg599"] Nov 03 16:17:58.002635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:17:58.002604 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caac17b6-2bd9-4903-ba9c-7fbbabac6870" path="/var/lib/kubelet/pods/caac17b6-2bd9-4903-ba9c-7fbbabac6870/volumes" Nov 03 16:18:08.992034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:08.991988 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:12.585116 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:18:12.585073 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:18:26.055820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:26.055783 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:26.390862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:26.390784 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:29.640448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:29.640411 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:29.848653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:29.848608 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:32.666996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:32.666956 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:32.865258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:32.865222 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:38.138069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.138037 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6"] Nov 03 16:18:38.138484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.138411 5946 cpu_manager.go:414] "containerMap: removing container" podUID="caac17b6-2bd9-4903-ba9c-7fbbabac6870" containerName="authorino" Nov 03 16:18:38.138484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.138422 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caac17b6-2bd9-4903-ba9c-7fbbabac6870" containerName="authorino" Nov 03 16:18:38.138557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.138501 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="caac17b6-2bd9-4903-ba9c-7fbbabac6870" containerName="authorino" Nov 03 16:18:38.140639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.140624 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.201108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.201071 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6"] Nov 03 16:18:38.228310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.228282 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnqv\" (UniqueName: \"kubernetes.io/projected/6fdb25a7-9065-46e7-b5ba-361ee9c1a10a-kube-api-access-wlnqv\") pod \"llmisvc-controller-manager-6466c554d8-tzxp6\" (UID: \"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.228473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.228349 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fdb25a7-9065-46e7-b5ba-361ee9c1a10a-cert\") pod \"llmisvc-controller-manager-6466c554d8-tzxp6\" (UID: \"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.329025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.328985 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fdb25a7-9065-46e7-b5ba-361ee9c1a10a-cert\") pod \"llmisvc-controller-manager-6466c554d8-tzxp6\" (UID: \"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.329192 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.329052 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnqv\" (UniqueName: \"kubernetes.io/projected/6fdb25a7-9065-46e7-b5ba-361ee9c1a10a-kube-api-access-wlnqv\") pod \"llmisvc-controller-manager-6466c554d8-tzxp6\" (UID: \"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.331708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.331680 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fdb25a7-9065-46e7-b5ba-361ee9c1a10a-cert\") pod \"llmisvc-controller-manager-6466c554d8-tzxp6\" (UID: \"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.391302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.391236 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnqv\" (UniqueName: \"kubernetes.io/projected/6fdb25a7-9065-46e7-b5ba-361ee9c1a10a-kube-api-access-wlnqv\") pod \"llmisvc-controller-manager-6466c554d8-tzxp6\" (UID: \"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.456852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.456816 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:38.835517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.835487 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6"] Nov 03 16:18:38.838078 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:18:38.838055 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdb25a7_9065_46e7_b5ba_361ee9c1a10a.slice/crio-9348d068d69a38684e2d8f4dfea29dbe940c656764cf150dffa8c166f15ab93d WatchSource:0}: Error finding container 9348d068d69a38684e2d8f4dfea29dbe940c656764cf150dffa8c166f15ab93d: Status 404 returned error can't find the container with id 9348d068d69a38684e2d8f4dfea29dbe940c656764cf150dffa8c166f15ab93d Nov 03 16:18:38.874423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:38.874395 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" event={"ID":"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a","Type":"ContainerStarted","Data":"9348d068d69a38684e2d8f4dfea29dbe940c656764cf150dffa8c166f15ab93d"} Nov 03 16:18:39.881451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:39.881362 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" event={"ID":"6fdb25a7-9065-46e7-b5ba-361ee9c1a10a","Type":"ContainerStarted","Data":"b1bcb6b8cc5f31567b4cd1d8d7ec997aeab981915ba65c5bbfc3b17c910502ec"} Nov 03 16:18:39.881903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:39.881497 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:18:40.011921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:40.011864 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" podStartSLOduration=1.293339245 podStartE2EDuration="2.011846919s" totalImagesPullingTime="718.507674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:40.009744802 +0000 UTC m=+390.130797626" watchObservedRunningTime="2026-11-03 16:18:40.011846919 +0000 UTC m=+390.132899743" Nov 03 16:18:42.584675 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:18:42.584639 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:18:50.013875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:50.013784 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:50.216672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:18:50.216639 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:10.888940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:10.888913 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-tzxp6" Nov 03 16:19:11.064014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.063985 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d"] Nov 03 16:19:11.064376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.064328 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" podUID="1eb79f6e-18f4-4fbe-9069-599dce82a523" containerName="manager" containerID="cri-o://6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757" gracePeriod=30 Nov 03 16:19:11.315369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.315347 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:19:11.425193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.425157 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb79f6e-18f4-4fbe-9069-599dce82a523-cert\" (UniqueName: \"kubernetes.io/secret/1eb79f6e-18f4-4fbe-9069-599dce82a523-cert\") pod \"1eb79f6e-18f4-4fbe-9069-599dce82a523\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " Nov 03 16:19:11.425364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.425233 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb79f6e-18f4-4fbe-9069-599dce82a523-kube-api-access-99wwf\" (UniqueName: \"kubernetes.io/projected/1eb79f6e-18f4-4fbe-9069-599dce82a523-kube-api-access-99wwf\") pod \"1eb79f6e-18f4-4fbe-9069-599dce82a523\" (UID: \"1eb79f6e-18f4-4fbe-9069-599dce82a523\") " Nov 03 16:19:11.427387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.427351 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb79f6e-18f4-4fbe-9069-599dce82a523-kube-api-access-99wwf" pod "1eb79f6e-18f4-4fbe-9069-599dce82a523" (UID: "1eb79f6e-18f4-4fbe-9069-599dce82a523"). InnerVolumeSpecName "kube-api-access-99wwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:11.427514 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.427440 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb79f6e-18f4-4fbe-9069-599dce82a523-cert" pod "1eb79f6e-18f4-4fbe-9069-599dce82a523" (UID: "1eb79f6e-18f4-4fbe-9069-599dce82a523"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:19:11.526560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.526526 5946 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eb79f6e-18f4-4fbe-9069-599dce82a523-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:19:11.526560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.526555 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wwf\" (UniqueName: \"kubernetes.io/projected/1eb79f6e-18f4-4fbe-9069-599dce82a523-kube-api-access-99wwf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:19:11.984451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:11.984408 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:12.040728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.040698 5946 generic.go:356] "Generic (PLEG): container finished" podID="1eb79f6e-18f4-4fbe-9069-599dce82a523" containerID="6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757" exitCode=0 Nov 03 16:19:12.040881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.040748 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" Nov 03 16:19:12.040881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.040791 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" event={"ID":"1eb79f6e-18f4-4fbe-9069-599dce82a523","Type":"ContainerDied","Data":"6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757"} Nov 03 16:19:12.040881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.040839 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d" event={"ID":"1eb79f6e-18f4-4fbe-9069-599dce82a523","Type":"ContainerDied","Data":"c56b375d51a6d85b1ce6a28828a938fdcdcbc3bbb42000ea4d0e67f3023e26ab"} Nov 03 16:19:12.040881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.040859 5946 scope.go:122] "RemoveContainer" containerID="6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757" Nov 03 16:19:12.049298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.049280 5946 scope.go:122] "RemoveContainer" containerID="6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757" Nov 03 16:19:12.049560 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:19:12.049542 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757\": container with ID starting with 6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757 not found: ID does not exist" containerID="6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757" Nov 03 16:19:12.049606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.049568 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757"} err="failed to get container status \"6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757\": rpc error: code = NotFound desc = could not find container \"6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757\": container with ID starting with 6ee102ea20a5d2714f3b7b4e7d8b329e300aa8e75535e9836f21a0e5e865c757 not found: ID does not exist" Nov 03 16:19:12.186354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.186325 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:12.212559 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.212531 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d"] Nov 03 16:19:12.223991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:12.223966 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmq6d"] Nov 03 16:19:12.584936 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:19:12.584892 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:19:14.002201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:14.002170 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb79f6e-18f4-4fbe-9069-599dce82a523" path="/var/lib/kubelet/pods/1eb79f6e-18f4-4fbe-9069-599dce82a523/volumes" Nov 03 16:19:26.391506 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:26.391471 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:33.078562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:33.078518 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:36.335798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:36.335747 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:19:37.630968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:37.630933 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:42.249787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:42.249754 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:42.585281 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:19:42.585192 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:19:45.670221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:45.670165 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:19:48.457672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:48.457636 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:54.134787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:54.134751 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:55.617983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:55.617954 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:56.816143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:56.816103 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:19:58.193154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:19:58.193124 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:02.263795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:02.263759 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:05.684237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:05.684187 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:08.930652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:08.930615 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:20:12.584496 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:20:12.584460 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:20:15.884512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:15.884479 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2"] Nov 03 16:20:15.884980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:15.884880 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1eb79f6e-18f4-4fbe-9069-599dce82a523" containerName="manager" Nov 03 16:20:15.884980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:15.884891 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb79f6e-18f4-4fbe-9069-599dce82a523" containerName="manager" Nov 03 16:20:15.884980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:15.884962 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb79f6e-18f4-4fbe-9069-599dce82a523" containerName="manager" Nov 03 16:20:15.888996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:15.888974 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.002789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.002762 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2"] Nov 03 16:20:16.022867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.022829 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.022872 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lch2d\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-kube-api-access-lch2d\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.022908 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.022990 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.023019 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023164 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.023063 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.023182 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72db61-3320-4470-b8cd-872c9d48b4f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.023292 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.023357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.023345 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.045076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.045044 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mw47n\"" Nov 03 16:20:16.124406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124374 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72db61-3320-4470-b8cd-872c9d48b4f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124424 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124460 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124479 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124500 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lch2d\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-kube-api-access-lch2d\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124523 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124551 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124579 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.124950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124609 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.125067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.124984 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.125176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.125150 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72db61-3320-4470-b8cd-872c9d48b4f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.125266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.125245 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.125316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.125266 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.125351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.125314 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.127133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.127102 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.127236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.127175 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.190352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.190285 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.190477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.190407 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lch2d\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-kube-api-access-lch2d\") pod \"maas-default-gateway-openshift-default-5bc884dd87-j5jx2\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.201278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.201253 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:16.647401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.647376 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2"] Nov 03 16:20:16.649076 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:20:16.649046 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac72db61_3320_4470_b8cd_872c9d48b4f4.slice/crio-71a3cdf6b93a656b00a324014c307cc88c3b73e6b41433c672cde8af64e9fbc0 WatchSource:0}: Error finding container 71a3cdf6b93a656b00a324014c307cc88c3b73e6b41433c672cde8af64e9fbc0: Status 404 returned error can't find the container with id 71a3cdf6b93a656b00a324014c307cc88c3b73e6b41433c672cde8af64e9fbc0 Nov 03 16:20:16.651186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.651156 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:20:16.651279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.651243 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:20:16.651279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:16.651274 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:20:17.353169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:17.353130 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" event={"ID":"ac72db61-3320-4470-b8cd-872c9d48b4f4","Type":"ContainerStarted","Data":"b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2"} Nov 03 16:20:17.353704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:17.353192 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" event={"ID":"ac72db61-3320-4470-b8cd-872c9d48b4f4","Type":"ContainerStarted","Data":"71a3cdf6b93a656b00a324014c307cc88c3b73e6b41433c672cde8af64e9fbc0"} Nov 03 16:20:17.535148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:17.535073 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podStartSLOduration=2.535054878 podStartE2EDuration="2.535054878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:17.53478844 +0000 UTC m=+487.655841263" watchObservedRunningTime="2026-11-03 16:20:17.535054878 +0000 UTC m=+487.656107701" Nov 03 16:20:18.175140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:18.175104 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:18.201685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:18.201649 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:18.203138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:18.203100 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-j5jx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:20:18.203283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:18.203161 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:20:19.184340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:19.184304 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:19.202590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:19.202559 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-j5jx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:20:19.202727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:19.202609 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:20:20.201720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:20.201684 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-j5jx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:20:20.202107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:20.201738 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:20:21.202688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:21.202653 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-j5jx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:20:21.203070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:21.202712 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:20:22.202644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:22.202605 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-j5jx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:20:22.202840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:22.202663 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:20:23.201605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:23.201563 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-j5jx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:20:23.201822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:23.201613 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:20:24.732170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:24.732139 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:24.732708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:24.732676 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:24.733545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:24.733524 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 16:20:25.140116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:25.140040 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:26.732984 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:26.732949 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:29.199586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:29.199552 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:29.578771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:29.578740 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:31.026847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.026812 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv"] Nov 03 16:20:31.030453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.030437 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.138807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.138767 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv"] Nov 03 16:20:31.170703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.170669 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.170862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.170713 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.170862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.170846 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.170941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.170876 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.170941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.170899 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83eaba29-3d02-41ab-bad7-c4d076ca6215-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.170941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.170914 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlck5\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-kube-api-access-xlck5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.170941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.170933 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.171063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.171020 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.171063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.171056 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272156 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272215 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272260 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272308 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272347 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272393 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272516 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272567 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83eaba29-3d02-41ab-bad7-c4d076ca6215-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272592 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlck5\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-kube-api-access-xlck5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272643 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.272691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.272660 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.273487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.273457 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.273777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.273511 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.273859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.273772 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83eaba29-3d02-41ab-bad7-c4d076ca6215-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.275309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.275284 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.275523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.275502 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.341758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.341677 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.341916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.341833 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlck5\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-kube-api-access-xlck5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-scnzv\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.343531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.343505 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:31.756156 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.756120 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd98557cb-dzhcq"] Nov 03 16:20:31.756377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.756315 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-bd98557cb-dzhcq" podUID="261c7408-c4e6-45df-8a13-ed9f9558e729" containerName="authorino" containerID="cri-o://5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b" gracePeriod=30 Nov 03 16:20:31.798357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.798327 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv"] Nov 03 16:20:31.848199 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:20:31.848168 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eaba29_3d02_41ab_bad7_c4d076ca6215.slice/crio-657d4fcfd72b8da0e3d3d7ee5613dde51f576abc3b7e8181dee4f013de07418d WatchSource:0}: Error finding container 657d4fcfd72b8da0e3d3d7ee5613dde51f576abc3b7e8181dee4f013de07418d: Status 404 returned error can't find the container with id 657d4fcfd72b8da0e3d3d7ee5613dde51f576abc3b7e8181dee4f013de07418d Nov 03 16:20:31.862470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.862439 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:20:31.862558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.862507 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:20:31.862558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:31.862535 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:20:32.015984 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.015958 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:20:32.181461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.181426 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261c7408-c4e6-45df-8a13-ed9f9558e729-kube-api-access-4tvcx\" (UniqueName: \"kubernetes.io/projected/261c7408-c4e6-45df-8a13-ed9f9558e729-kube-api-access-4tvcx\") pod \"261c7408-c4e6-45df-8a13-ed9f9558e729\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " Nov 03 16:20:32.181854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.181597 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261c7408-c4e6-45df-8a13-ed9f9558e729-tls-cert\" (UniqueName: \"kubernetes.io/secret/261c7408-c4e6-45df-8a13-ed9f9558e729-tls-cert\") pod \"261c7408-c4e6-45df-8a13-ed9f9558e729\" (UID: \"261c7408-c4e6-45df-8a13-ed9f9558e729\") " Nov 03 16:20:32.183761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.183733 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c7408-c4e6-45df-8a13-ed9f9558e729-kube-api-access-4tvcx" pod "261c7408-c4e6-45df-8a13-ed9f9558e729" (UID: "261c7408-c4e6-45df-8a13-ed9f9558e729"). InnerVolumeSpecName "kube-api-access-4tvcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:20:32.192830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.192802 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261c7408-c4e6-45df-8a13-ed9f9558e729-tls-cert" pod "261c7408-c4e6-45df-8a13-ed9f9558e729" (UID: "261c7408-c4e6-45df-8a13-ed9f9558e729"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:20:32.282805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.282723 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tvcx\" (UniqueName: \"kubernetes.io/projected/261c7408-c4e6-45df-8a13-ed9f9558e729-kube-api-access-4tvcx\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:20:32.282805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.282754 5946 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261c7408-c4e6-45df-8a13-ed9f9558e729-tls-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:20:32.432918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.432877 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" event={"ID":"83eaba29-3d02-41ab-bad7-c4d076ca6215","Type":"ContainerStarted","Data":"0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142"} Nov 03 16:20:32.433185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.433169 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" event={"ID":"83eaba29-3d02-41ab-bad7-c4d076ca6215","Type":"ContainerStarted","Data":"657d4fcfd72b8da0e3d3d7ee5613dde51f576abc3b7e8181dee4f013de07418d"} Nov 03 16:20:32.435845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.435815 5946 generic.go:356] "Generic (PLEG): container finished" podID="261c7408-c4e6-45df-8a13-ed9f9558e729" containerID="5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b" exitCode=0 Nov 03 16:20:32.436089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.436068 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd98557cb-dzhcq" event={"ID":"261c7408-c4e6-45df-8a13-ed9f9558e729","Type":"ContainerDied","Data":"5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b"} Nov 03 16:20:32.436246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.436226 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd98557cb-dzhcq" event={"ID":"261c7408-c4e6-45df-8a13-ed9f9558e729","Type":"ContainerDied","Data":"9912059d5d57a4f62f770917e863bcb844503ca32ed3b99f501f08c55f3533d9"} Nov 03 16:20:32.436359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.436346 5946 scope.go:122] "RemoveContainer" containerID="5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b" Nov 03 16:20:32.436600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.436585 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd98557cb-dzhcq" Nov 03 16:20:32.448115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.448093 5946 scope.go:122] "RemoveContainer" containerID="5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b" Nov 03 16:20:32.448465 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:20:32.448443 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b\": container with ID starting with 5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b not found: ID does not exist" containerID="5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b" Nov 03 16:20:32.448521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.448478 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b"} err="failed to get container status \"5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b\": rpc error: code = NotFound desc = could not find container \"5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b\": container with ID starting with 5514f0f40400161282be6e69086d71819aaa393600fe82b4cf8591c023a2b41b not found: ID does not exist" Nov 03 16:20:32.558847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.558727 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podStartSLOduration=2.558712029 podStartE2EDuration="2.558712029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:32.557463311 +0000 UTC m=+502.678516125" watchObservedRunningTime="2026-11-03 16:20:32.558712029 +0000 UTC m=+502.679764851" Nov 03 16:20:32.673440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.673408 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd98557cb-dzhcq"] Nov 03 16:20:32.681963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:32.681933 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bd98557cb-dzhcq"] Nov 03 16:20:33.344645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:33.344606 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:33.346119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:33.346082 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:20:33.346298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:33.346148 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:20:34.002445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:34.002415 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261c7408-c4e6-45df-8a13-ed9f9558e729" path="/var/lib/kubelet/pods/261c7408-c4e6-45df-8a13-ed9f9558e729/volumes" Nov 03 16:20:34.344230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:34.344120 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:20:34.344373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:34.344233 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:20:35.344709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:35.344671 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:20:35.345140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:35.344725 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:20:36.344917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:36.344882 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:20:36.345364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:36.344938 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:20:37.344304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:37.344263 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:20:37.344498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:37.344330 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:20:38.344119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:38.344084 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:20:38.344663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:38.344142 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:20:39.297894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:39.297816 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:39.345030 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:39.344993 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:20:39.345551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:39.345065 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:20:39.583532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:39.583403 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:41.385403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:41.385365 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:20:41.385878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:41.385436 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:42.427366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:42.427330 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-scnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:20:42.427787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:42.427396 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:42.486135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:42.486106 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:42.486510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:42.486490 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:42.487395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:42.487376 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:20:42.584890 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:20:42.584856 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:20:44.770884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:44.770854 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:45.817622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:45.817590 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:50.344396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:20:50.344345 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:03.620231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:03.620159 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:04.148711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:04.148681 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:05.794834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:05.794791 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:06.081827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:06.081751 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:07.565198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:07.565167 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:07.862479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:07.862399 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:12.584900 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:21:12.584858 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:21:27.064847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:27.064794 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:27.417425 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:27.417291 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:28.345138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:28.345092 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:28.548720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:28.548679 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:42.584597 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:21:42.584558 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:21:54.025181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:54.025102 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:54.221810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:54.221774 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:54.688919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:54.688890 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:54.920238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:54.920189 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:55.122323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.122289 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:55.314719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.314688 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:55.509620 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.509585 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:55.532009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.531978 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g"] Nov 03 16:21:55.532445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.532431 5946 cpu_manager.go:414] "containerMap: removing container" podUID="261c7408-c4e6-45df-8a13-ed9f9558e729" containerName="authorino" Nov 03 16:21:55.532505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.532448 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c7408-c4e6-45df-8a13-ed9f9558e729" containerName="authorino" Nov 03 16:21:55.532853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.532827 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c7408-c4e6-45df-8a13-ed9f9558e729" containerName="authorino" Nov 03 16:21:55.537093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.537067 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.593317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.593292 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-192564-openshift-default-dockercfg-256th\"" Nov 03 16:21:55.644557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.644519 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g"] Nov 03 16:21:55.649023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649001 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61028362-d71d-4d86-a1b6-7dca6553c61d-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649036 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649072 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649172 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649198 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649262 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649355 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649386 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drpv4\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-kube-api-access-drpv4\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.649504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.649490 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.706132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.706104 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:55.750114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750080 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750126 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750170 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750193 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750248 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750322 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750349 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpv4\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-kube-api-access-drpv4\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750407 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750450 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61028362-d71d-4d86-a1b6-7dca6553c61d-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750729 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.750865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750835 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.751010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.750992 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.751082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.751016 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.751389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.751363 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61028362-d71d-4d86-a1b6-7dca6553c61d-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.752739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.752719 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.753083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.753063 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.811330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.811257 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.811485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.811462 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drpv4\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-kube-api-access-drpv4\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.853105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.853071 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:55.907444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:55.907372 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:56.101908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.101827 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:56.297773 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.297743 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:56.330922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.330881 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g"] Nov 03 16:21:56.331713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.331681 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:21:56.331823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.331742 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:21:56.331823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.331774 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:21:56.494409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.494368 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:56.693879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.693794 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:56.857795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.857760 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" event={"ID":"61028362-d71d-4d86-a1b6-7dca6553c61d","Type":"ContainerStarted","Data":"2be03d1904f66e99730ea4b1566b3eebe8c4c2e6456636a5415119ca7aec7b9e"} Nov 03 16:21:56.857963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.857808 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" event={"ID":"61028362-d71d-4d86-a1b6-7dca6553c61d","Type":"ContainerStarted","Data":"f024f7daa75ae85b7dfd19a108af8b71be591492f539e5e4d26a0177d9c24333"} Nov 03 16:21:56.984869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:56.984802 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podStartSLOduration=1.984780853 podStartE2EDuration="1.984780853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:56.98219066 +0000 UTC m=+587.103243497" watchObservedRunningTime="2026-11-03 16:21:56.984780853 +0000 UTC m=+587.105833680" Nov 03 16:21:57.853301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:57.853271 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:57.858953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:57.858930 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:57.862652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:57.862626 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:57.863433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:57.863417 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:21:59.145142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:59.145096 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:59.363248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:59.363200 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:59.779550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:59.779516 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:59.983031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:21:59.983004 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:00.520635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:00.520583 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:00.774405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:00.774310 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:00.970684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:00.970639 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:01.186535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:01.186442 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:01.396900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:01.396865 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:01.798771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:01.798738 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:01.998860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:01.998828 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:03.997996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:03.997955 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:04.190815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.190779 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:04.392986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.392897 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:04.501679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.501646 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d"] Nov 03 16:22:04.505310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.505295 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.563928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.563035 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-547a38-openshift-default-dockercfg-tnnkl\"" Nov 03 16:22:04.615287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.615252 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d"] Nov 03 16:22:04.621703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.621679 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:04.638125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638092 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638127 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638149 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638301 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638329 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638351 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27c94b-e6be-42e4-b29a-25b216cf8065-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638448 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638505 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9lf\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-kube-api-access-kq9lf\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.638578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.638548 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740299 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740369 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740405 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740451 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740481 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740514 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27c94b-e6be-42e4-b29a-25b216cf8065-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740560 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740609 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9lf\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-kube-api-access-kq9lf\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.740797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.740663 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.741254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.741192 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.744481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.744452 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.745132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.745105 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27c94b-e6be-42e4-b29a-25b216cf8065-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.745420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.745397 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.746065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.746041 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.746924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.746899 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.751271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.751247 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.807000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.806959 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9lf\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-kube-api-access-kq9lf\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.807713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.807693 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.819751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.819716 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:04.832976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:04.832941 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:05.031887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.031791 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:05.303815 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:22:05.303778 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c27c94b_e6be_42e4_b29a_25b216cf8065.slice/crio-8163c448292bd46b3e91b735e75d385ebf87c44312bffe4656ef56c35b79db65 WatchSource:0}: Error finding container 8163c448292bd46b3e91b735e75d385ebf87c44312bffe4656ef56c35b79db65: Status 404 returned error can't find the container with id 8163c448292bd46b3e91b735e75d385ebf87c44312bffe4656ef56c35b79db65 Nov 03 16:22:05.305056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.305033 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d"] Nov 03 16:22:05.305968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.305933 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:05.306066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.306001 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:05.306066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.306031 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:05.442396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.442365 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:05.631399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.631306 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:05.914957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.914852 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" event={"ID":"7c27c94b-e6be-42e4-b29a-25b216cf8065","Type":"ContainerStarted","Data":"ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778"} Nov 03 16:22:05.914957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:05.914922 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" event={"ID":"7c27c94b-e6be-42e4-b29a-25b216cf8065","Type":"ContainerStarted","Data":"8163c448292bd46b3e91b735e75d385ebf87c44312bffe4656ef56c35b79db65"} Nov 03 16:22:06.040037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:06.039978 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podStartSLOduration=2.039965474 podStartE2EDuration="2.039965474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:06.039298663 +0000 UTC m=+596.160351486" watchObservedRunningTime="2026-11-03 16:22:06.039965474 +0000 UTC m=+596.161018370" Nov 03 16:22:06.820262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:06.820229 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:06.825480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:06.825450 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:06.919613 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:06.919579 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:06.920601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:06.920581 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:22:09.155222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:09.155172 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:09.360161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:09.360125 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:10.068123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:10.068098 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:22:10.068932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:10.068908 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:22:10.720446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:10.720413 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:10.911006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:10.910961 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:11.106776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:11.106750 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:11.801612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:11.801583 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:11.993121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:11.993090 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:12.584928 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:22:12.584891 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:22:14.441471 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:14.441417 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:14.644197 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:14.644150 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:15.605803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:15.605771 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:15.808731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:15.808700 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:16.005813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:16.005785 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:16.199002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:16.198975 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:18.408564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:18.408526 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:22:19.504151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:19.504117 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:19.702250 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:19.702194 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:21.265378 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:21.265345 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:21.461066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:21.461035 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:21.649923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:21.649836 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:24.601662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:24.601620 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:24.791788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:24.791755 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:25.070618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.070579 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg"] Nov 03 16:22:25.074009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.073991 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.134190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.134161 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg"] Nov 03 16:22:25.140201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.140162 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbd6r\" (UniqueName: \"kubernetes.io/projected/157d399c-c209-41dd-b4e4-b4e3cafdc574-kube-api-access-vbd6r\") pod \"payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.140398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.140349 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157d399c-c209-41dd-b4e4-b4e3cafdc574-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.182946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.182904 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-bcd756\"" Nov 03 16:22:25.182946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.182911 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-bcd756-dockercfg-7zthm\"" Nov 03 16:22:25.241169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.241138 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157d399c-c209-41dd-b4e4-b4e3cafdc574-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.241365 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.241222 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6r\" (UniqueName: \"kubernetes.io/projected/157d399c-c209-41dd-b4e4-b4e3cafdc574-kube-api-access-vbd6r\") pod \"payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.241854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.241834 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157d399c-c209-41dd-b4e4-b4e3cafdc574-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.302171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.302144 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbd6r\" (UniqueName: \"kubernetes.io/projected/157d399c-c209-41dd-b4e4-b4e3cafdc574-kube-api-access-vbd6r\") pod \"payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.387415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.387341 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:25.766043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.766020 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg"] Nov 03 16:22:25.768049 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:22:25.768024 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157d399c_c209_41dd_b4e4_b4e3cafdc574.slice/crio-2b40acd64f96b7416a306b16a1597dd8901ec7c12ee75bbed91be89551250a81 WatchSource:0}: Error finding container 2b40acd64f96b7416a306b16a1597dd8901ec7c12ee75bbed91be89551250a81: Status 404 returned error can't find the container with id 2b40acd64f96b7416a306b16a1597dd8901ec7c12ee75bbed91be89551250a81 Nov 03 16:22:25.770009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:25.769990 5946 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:22:26.020610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:26.020519 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" event={"ID":"157d399c-c209-41dd-b4e4-b4e3cafdc574","Type":"ContainerStarted","Data":"2b40acd64f96b7416a306b16a1597dd8901ec7c12ee75bbed91be89551250a81"} Nov 03 16:22:26.504535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:26.504505 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:27.383087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:27.383055 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:28.183390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:28.183357 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:22:29.045086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:29.045042 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" event={"ID":"157d399c-c209-41dd-b4e4-b4e3cafdc574","Type":"ContainerStarted","Data":"0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6"} Nov 03 16:22:29.045563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:29.045230 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:29.165914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:29.165863 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" podStartSLOduration=1.360392472 podStartE2EDuration="4.165847172s" totalImagesPullingTime="2.8054547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:29.164752975 +0000 UTC m=+619.285805799" watchObservedRunningTime="2026-11-03 16:22:29.165847172 +0000 UTC m=+619.286900037" Nov 03 16:22:29.593448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:29.593419 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:29.799281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:29.799249 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:29.984466 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:29.984436 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj"] Nov 03 16:22:29.988131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:29.988102 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.048340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.048308 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj"] Nov 03 16:22:30.089245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.089194 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad44a8d-f155-4329-afd7-ceaed3489ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.089393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.089257 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrmj\" (UniqueName: \"kubernetes.io/projected/aad44a8d-f155-4329-afd7-ceaed3489ede-kube-api-access-cwrmj\") pod \"payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.096174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.096155 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-547a38\"" Nov 03 16:22:30.096245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.096155 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-547a38-dockercfg-2dcx9\"" Nov 03 16:22:30.150195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.150165 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4"] Nov 03 16:22:30.153740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.153723 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.182157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.182126 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4"] Nov 03 16:22:30.190999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.190756 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5078901f-7793-4090-8611-36726bb133c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.190999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.190799 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6cx\" (UniqueName: \"kubernetes.io/projected/5078901f-7793-4090-8611-36726bb133c7-kube-api-access-vm6cx\") pod \"payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.191967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.191034 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad44a8d-f155-4329-afd7-ceaed3489ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.191967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.191109 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrmj\" (UniqueName: \"kubernetes.io/projected/aad44a8d-f155-4329-afd7-ceaed3489ede-kube-api-access-cwrmj\") pod \"payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.191967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.191881 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad44a8d-f155-4329-afd7-ceaed3489ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.251791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.251714 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrmj\" (UniqueName: \"kubernetes.io/projected/aad44a8d-f155-4329-afd7-ceaed3489ede-kube-api-access-cwrmj\") pod \"payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.291723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.291692 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5078901f-7793-4090-8611-36726bb133c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.291884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.291728 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6cx\" (UniqueName: \"kubernetes.io/projected/5078901f-7793-4090-8611-36726bb133c7-kube-api-access-vm6cx\") pod \"payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.292521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.292494 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5078901f-7793-4090-8611-36726bb133c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.300943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.300924 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:30.352552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.352524 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6cx\" (UniqueName: \"kubernetes.io/projected/5078901f-7793-4090-8611-36726bb133c7-kube-api-access-vm6cx\") pod \"payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.467894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.467857 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:30.708282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.708249 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj"] Nov 03 16:22:30.710617 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:22:30.710577 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad44a8d_f155_4329_afd7_ceaed3489ede.slice/crio-2e9a3376eefbfc7caeb1e7c7949e138eec1c5d58a91141dce66590df11ba42d4 WatchSource:0}: Error finding container 2e9a3376eefbfc7caeb1e7c7949e138eec1c5d58a91141dce66590df11ba42d4: Status 404 returned error can't find the container with id 2e9a3376eefbfc7caeb1e7c7949e138eec1c5d58a91141dce66590df11ba42d4 Nov 03 16:22:30.862072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:30.862021 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4"] Nov 03 16:22:30.864372 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:22:30.864343 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5078901f_7793_4090_8611_36726bb133c7.slice/crio-5e791006a5c564784f7a53f714f0c6ddd8dd45cec4251b7f9187e07c6b1a72fb WatchSource:0}: Error finding container 5e791006a5c564784f7a53f714f0c6ddd8dd45cec4251b7f9187e07c6b1a72fb: Status 404 returned error can't find the container with id 5e791006a5c564784f7a53f714f0c6ddd8dd45cec4251b7f9187e07c6b1a72fb Nov 03 16:22:31.058017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.057974 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" event={"ID":"5078901f-7793-4090-8611-36726bb133c7","Type":"ContainerStarted","Data":"a9053269f29c18093e25910de20daf304aaed92e383433297d9aad9a22aa1c5d"} Nov 03 16:22:31.058505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.058044 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" event={"ID":"5078901f-7793-4090-8611-36726bb133c7","Type":"ContainerStarted","Data":"5e791006a5c564784f7a53f714f0c6ddd8dd45cec4251b7f9187e07c6b1a72fb"} Nov 03 16:22:31.058505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.058143 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:31.059634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.059602 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" event={"ID":"aad44a8d-f155-4329-afd7-ceaed3489ede","Type":"ContainerStarted","Data":"71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e"} Nov 03 16:22:31.059758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.059648 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" event={"ID":"aad44a8d-f155-4329-afd7-ceaed3489ede","Type":"ContainerStarted","Data":"2e9a3376eefbfc7caeb1e7c7949e138eec1c5d58a91141dce66590df11ba42d4"} Nov 03 16:22:31.059911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.059897 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:31.178655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.178555 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" podStartSLOduration=1.1785399 podStartE2EDuration="1.1785399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:31.17794566 +0000 UTC m=+621.298998481" watchObservedRunningTime="2026-11-03 16:22:31.1785399 +0000 UTC m=+621.299592734" Nov 03 16:22:31.311095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.311015 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" podStartSLOduration=2.31100074 podStartE2EDuration="2.31100074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:31.310128616 +0000 UTC m=+621.431181439" watchObservedRunningTime="2026-11-03 16:22:31.31100074 +0000 UTC m=+621.432053563" Nov 03 16:22:31.803681 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:31.803649 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:32.010108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:32.010082 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:32.204856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:32.204782 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:36.982068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:36.982014 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:37.091025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:37.090989 5946 generic.go:356] "Generic (PLEG): container finished" podID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerID="2be03d1904f66e99730ea4b1566b3eebe8c4c2e6456636a5415119ca7aec7b9e" exitCode=0 Nov 03 16:22:37.091195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:37.091033 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" event={"ID":"61028362-d71d-4d86-a1b6-7dca6553c61d","Type":"ContainerDied","Data":"2be03d1904f66e99730ea4b1566b3eebe8c4c2e6456636a5415119ca7aec7b9e"} Nov 03 16:22:37.091443 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:37.091425 5946 scope.go:122] "RemoveContainer" containerID="2be03d1904f66e99730ea4b1566b3eebe8c4c2e6456636a5415119ca7aec7b9e" Nov 03 16:22:37.092677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:37.092650 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:37.092731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:37.092721 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:37.092765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:37.092751 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:38.097253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:38.097220 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" event={"ID":"61028362-d71d-4d86-a1b6-7dca6553c61d","Type":"ContainerStarted","Data":"e2d5037e013f215c4104cf379cf27c002579dae31c1847d4a0e30c1f826b07d5"} Nov 03 16:22:38.509755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:38.509723 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:22:38.853791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:38.853704 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:22:38.854833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:38.854808 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 16:22:38.854965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:38.854860 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 16:22:39.853887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:39.853850 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 16:22:39.854305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:39.853937 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 16:22:40.054668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:40.054642 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 16:22:40.853820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:40.853784 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 16:22:40.853996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:40.853840 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 16:22:41.786900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:41.786869 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:41.854136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:41.854101 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 16:22:41.854560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:41.854152 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 16:22:41.983138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:41.983101 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:42.067936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:42.067866 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:22:42.067936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:42.067933 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:22:42.193218 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:42.193171 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:42.585362 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:22:42.585319 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:22:42.853842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:42.853756 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 16:22:42.853842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:42.853814 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 16:22:42.863063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:42.863036 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:22:43.858573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:43.858545 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:22:43.859659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:43.859642 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:22:44.705024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:44.704932 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:47.057280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:47.057222 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:48.622309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:48.622272 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:22:52.254467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:52.254396 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:52.457701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:52.457646 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:52.652115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:52.652006 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:55.324910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:55.324874 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:55.518407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:55.518322 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:55.707308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:55.707277 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:56.233279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:56.233249 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:56.430172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:56.430127 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:57.625150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:57.625111 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:59.231837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.231804 5946 generic.go:356] "Generic (PLEG): container finished" podID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerID="ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778" exitCode=0 Nov 03 16:22:59.232319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.231856 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" event={"ID":"7c27c94b-e6be-42e4-b29a-25b216cf8065","Type":"ContainerDied","Data":"ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778"} Nov 03 16:22:59.232319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.232200 5946 scope.go:122] "RemoveContainer" containerID="ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778" Nov 03 16:22:59.233641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.233615 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:59.233712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.233684 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:59.233748 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.233717 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:22:59.364658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.364622 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9"] Nov 03 16:22:59.368006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.367974 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.420464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.420425 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432474 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-tmp-dir\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432546 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28625d13-a0fa-4bad-919b-3c325c396db2-tls-certs\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432588 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-dshm\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432618 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6ae690\"/\"default-dockercfg-w9645\"" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432477 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6ae690\"/\"openshift-service-ca.crt\"" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432715 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6ae690\"/\"kube-root-ca.crt\"" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432621 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-model-cache\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432792 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhhjf\" (UniqueName: \"kubernetes.io/projected/28625d13-a0fa-4bad-919b-3c325c396db2-kube-api-access-bhhjf\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432813 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6ae690\"/\"simulator-w5-6ae690-kserve-self-signed-certs\"" Nov 03 16:22:59.433457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.432826 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-home\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.435562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.435532 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9"] Nov 03 16:22:59.504248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.504108 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:59.533793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.533760 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-tmp-dir\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.533987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.533806 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28625d13-a0fa-4bad-919b-3c325c396db2-tls-certs\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.533987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.533834 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-dshm\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.533987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.533855 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-model-cache\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.533987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.533885 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhhjf\" (UniqueName: \"kubernetes.io/projected/28625d13-a0fa-4bad-919b-3c325c396db2-kube-api-access-bhhjf\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.533987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.533911 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-home\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.534303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.534193 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-tmp-dir\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.534366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.534309 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-model-cache\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.534612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.534588 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-home\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.536777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.536749 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-dshm\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.537311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.537289 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28625d13-a0fa-4bad-919b-3c325c396db2-tls-certs\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.594769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.594726 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhhjf\" (UniqueName: \"kubernetes.io/projected/28625d13-a0fa-4bad-919b-3c325c396db2-kube-api-access-bhhjf\") pod \"simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:22:59.679398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:22:59.679354 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:23:00.096747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:00.096718 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9"] Nov 03 16:23:00.099147 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:23:00.099122 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28625d13_a0fa_4bad_919b_3c325c396db2.slice/crio-f400b4be83964f112fd2498e27c6a0a4f1dba9b9b4eb0fbf3462331de5a20d28 WatchSource:0}: Error finding container f400b4be83964f112fd2498e27c6a0a4f1dba9b9b4eb0fbf3462331de5a20d28: Status 404 returned error can't find the container with id f400b4be83964f112fd2498e27c6a0a4f1dba9b9b4eb0fbf3462331de5a20d28 Nov 03 16:23:00.243863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:00.243826 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" event={"ID":"7c27c94b-e6be-42e4-b29a-25b216cf8065","Type":"ContainerStarted","Data":"d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e"} Nov 03 16:23:00.245150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:00.245124 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" event={"ID":"28625d13-a0fa-4bad-919b-3c325c396db2","Type":"ContainerStarted","Data":"f400b4be83964f112fd2498e27c6a0a4f1dba9b9b4eb0fbf3462331de5a20d28"} Nov 03 16:23:00.832655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:00.832619 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:23:00.833009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:00.832986 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 16:23:00.833150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:00.833125 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 16:23:01.826234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:01.821622 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 16:23:01.826234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:01.821685 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 16:23:02.561882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:02.561820 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:02.836452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:02.836348 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 16:23:02.837056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:02.837016 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 16:23:03.270266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:03.270228 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" event={"ID":"28625d13-a0fa-4bad-919b-3c325c396db2","Type":"ContainerStarted","Data":"08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01"} Nov 03 16:23:03.372772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:03.372738 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:03.401158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:03.401106 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podStartSLOduration=1.854825488 podStartE2EDuration="4.401092974s" totalImagesPullingTime="2.546267486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:03.400781303 +0000 UTC m=+653.521834126" watchObservedRunningTime="2026-11-03 16:23:03.401092974 +0000 UTC m=+653.522145835" Nov 03 16:23:03.572668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:03.572552 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:03.679767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:03.679735 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:23:03.833292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:03.826611 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 16:23:03.833292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:03.826675 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 16:23:04.835233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:04.826589 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 16:23:04.835233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:04.826652 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 16:23:05.824917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:05.824883 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:23:05.825242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:05.825222 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:23:05.826054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:05.826040 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:23:06.140768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:06.140633 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:06.350823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:06.350792 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:06.794688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:06.794652 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:06.997893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:06.997859 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:07.871078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:07.871043 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:09.679912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:09.679876 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:23:09.693000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:09.692973 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:23:10.319761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:10.319734 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:23:10.729845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:10.729761 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:23:12.588462 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:23:12.588418 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:23:13.065334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:13.065303 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:17.851036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:17.851000 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:20.712388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:20.712357 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:21.373919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:21.373849 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:21.572747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:21.572682 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:21.989082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:21.989051 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:22.188162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:22.188128 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:23.080350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:23.080314 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:27.735489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:27.735424 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:27.917716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:27.917688 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:29.981002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:29.980968 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:30.541442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.541397 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:30.744652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.744610 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9"] Nov 03 16:23:30.747497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.747481 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.822859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.822789 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"openshift-service-ca.crt\"" Nov 03 16:23:30.852995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.852961 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"kube-root-ca.crt\"" Nov 03 16:23:30.853167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.853024 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"default-dockercfg-vxsb2\"" Nov 03 16:23:30.853324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.853303 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"simulator-w3-192564-kserve-self-signed-certs\"" Nov 03 16:23:30.855569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.855543 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9"] Nov 03 16:23:30.859220 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.859187 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e310e79e-049f-407b-82f7-bcd67c882fa8-tls-certs\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.859286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.859268 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rlb\" (UniqueName: \"kubernetes.io/projected/e310e79e-049f-407b-82f7-bcd67c882fa8-kube-api-access-79rlb\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.859325 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.859298 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-tmp-dir\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.859362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.859337 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-model-cache\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.859430 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.859417 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-home\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.859521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.859503 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-dshm\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.883510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.883480 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:30.960947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.960916 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-dshm\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.960971 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e310e79e-049f-407b-82f7-bcd67c882fa8-tls-certs\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.961028 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rlb\" (UniqueName: \"kubernetes.io/projected/e310e79e-049f-407b-82f7-bcd67c882fa8-kube-api-access-79rlb\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.961056 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-tmp-dir\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.961100 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-model-cache\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.961149 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-home\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.961484 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-tmp-dir\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.961514 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-home\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.961757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.961569 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-model-cache\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.963554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.963514 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-dshm\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:30.963729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:30.963710 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e310e79e-049f-407b-82f7-bcd67c882fa8-tls-certs\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:31.028383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:31.028348 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rlb\" (UniqueName: \"kubernetes.io/projected/e310e79e-049f-407b-82f7-bcd67c882fa8-kube-api-access-79rlb\") pod \"simulator-w3-192564-kserve-567d666b65-qmgk9\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:31.058150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:31.058115 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:31.131357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:31.131176 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:31.324600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:31.324509 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:31.448167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:31.448146 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9"] Nov 03 16:23:31.450024 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:23:31.449999 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode310e79e_049f_407b_82f7_bcd67c882fa8.slice/crio-c7926b7e1d8e67ad811bd168fbd8ecb6f720799c27a60b8a42d3419ba0f25c72 WatchSource:0}: Error finding container c7926b7e1d8e67ad811bd168fbd8ecb6f720799c27a60b8a42d3419ba0f25c72: Status 404 returned error can't find the container with id c7926b7e1d8e67ad811bd168fbd8ecb6f720799c27a60b8a42d3419ba0f25c72 Nov 03 16:23:32.432323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:32.432287 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" event={"ID":"e310e79e-049f-407b-82f7-bcd67c882fa8","Type":"ContainerStarted","Data":"aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d"} Nov 03 16:23:32.432785 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:32.432336 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" event={"ID":"e310e79e-049f-407b-82f7-bcd67c882fa8","Type":"ContainerStarted","Data":"c7926b7e1d8e67ad811bd168fbd8ecb6f720799c27a60b8a42d3419ba0f25c72"} Nov 03 16:23:32.554533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:32.554478 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podStartSLOduration=2.554463563 podStartE2EDuration="2.554463563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:32.553623469 +0000 UTC m=+682.674676292" watchObservedRunningTime="2026-11-03 16:23:32.554463563 +0000 UTC m=+682.675516386" Nov 03 16:23:33.010994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:33.010956 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:33.058630 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:33.058592 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:36.533426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:36.533387 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:23:36.893939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:36.893836 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54"] Nov 03 16:23:36.898634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:36.898603 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:36.954870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:36.954822 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"simulator-w0-42da0f-kserve-self-signed-certs\"" Nov 03 16:23:37.005809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.005773 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"default-dockercfg-jqwqn\"" Nov 03 16:23:37.005994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.005881 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"kube-root-ca.crt\"" Nov 03 16:23:37.005994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.005895 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"openshift-service-ca.crt\"" Nov 03 16:23:37.007823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.007791 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54"] Nov 03 16:23:37.015829 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.015788 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:37.024261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.022969 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-home\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.024261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.023033 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tls-certs\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.024261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.023233 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-dshm\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.024261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.023317 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tmp-dir\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.024261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.023356 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-model-cache\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.024261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.023389 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrtv\" (UniqueName: \"kubernetes.io/projected/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-kube-api-access-frrtv\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.124533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124490 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tmp-dir\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.124735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124572 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-model-cache\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.124735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124604 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrtv\" (UniqueName: \"kubernetes.io/projected/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-kube-api-access-frrtv\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.124869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124771 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-home\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.124869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124846 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tls-certs\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.124971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124922 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-dshm\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.124971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124954 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tmp-dir\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.125083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.124971 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-model-cache\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.125248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.125225 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-home\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.127373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.127343 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-dshm\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.127621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.127602 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tls-certs\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.186381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.186301 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrtv\" (UniqueName: \"kubernetes.io/projected/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-kube-api-access-frrtv\") pod \"simulator-w0-42da0f-kserve-6cc86cd8f8-bng54\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.210185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.210147 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:37.596196 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.596173 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54"] Nov 03 16:23:37.598341 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:23:37.598311 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36af74ee_e4d9_4ec5_bbc1_71de63e9d800.slice/crio-4a6a7cffced73e5181950d9ff591395e0659e25c19c3df1aeafa94b5a4c72be7 WatchSource:0}: Error finding container 4a6a7cffced73e5181950d9ff591395e0659e25c19c3df1aeafa94b5a4c72be7: Status 404 returned error can't find the container with id 4a6a7cffced73e5181950d9ff591395e0659e25c19c3df1aeafa94b5a4c72be7 Nov 03 16:23:37.768487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:37.768454 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:38.463342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:38.463303 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" event={"ID":"36af74ee-e4d9-4ec5-bbc1-71de63e9d800","Type":"ContainerStarted","Data":"9e9b9d6226de65ef9f893fbc2efc07f185736ffcdc015d36de8ba1d05d4626b4"} Nov 03 16:23:38.463342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:38.463346 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" event={"ID":"36af74ee-e4d9-4ec5-bbc1-71de63e9d800","Type":"ContainerStarted","Data":"4a6a7cffced73e5181950d9ff591395e0659e25c19c3df1aeafa94b5a4c72be7"} Nov 03 16:23:38.585015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:38.584959 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podStartSLOduration=2.584937579 podStartE2EDuration="2.584937579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:38.583030969 +0000 UTC m=+688.704083793" watchObservedRunningTime="2026-11-03 16:23:38.584937579 +0000 UTC m=+688.705990423" Nov 03 16:23:39.211056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:39.211020 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:41.058922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:41.058883 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:41.071580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:41.071554 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:41.490372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:41.490340 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:23:42.580447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:42.580413 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:42.590501 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:23:42.590466 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:23:42.774724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:42.774698 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:43.177526 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:43.177489 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:43.378764 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:43.378731 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:47.210384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:47.210345 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:47.222913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:47.222890 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:47.468305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:47.468157 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:47.524844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:47.524816 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:23:51.290481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:51.290447 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:23:52.311901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:52.311866 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:54.585800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:54.585769 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:57.007882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:23:57.007837 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:01.756826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:01.756783 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:06.432358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:06.432325 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:06.625280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:06.625247 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:11.275792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:11.275743 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:11.861411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:11.861376 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:24:12.584732 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:24:12.584694 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:24:14.737194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.737163 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp"] Nov 03 16:24:14.743768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.743731 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:14.801640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.801608 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp"] Nov 03 16:24:14.837592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.837543 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"default-dockercfg-2nfhm\"" Nov 03 16:24:14.837754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.837618 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"simulator-w6-547a38-kserve-self-signed-certs\"" Nov 03 16:24:14.849818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.849796 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"openshift-service-ca.crt\"" Nov 03 16:24:14.849818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.849804 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"kube-root-ca.crt\"" Nov 03 16:24:14.873529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.873489 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:14.904747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.904714 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-tmp-dir\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:14.904943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.904845 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-home\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:14.904943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.904879 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbpkf\" (UniqueName: \"kubernetes.io/projected/7843122b-449a-4188-8755-b49147f6f8ed-kube-api-access-tbpkf\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:14.905072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.905050 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-model-cache\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:14.905108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.905088 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-dshm\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:14.905178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:14.905164 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7843122b-449a-4188-8755-b49147f6f8ed-tls-certs\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.006674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.006588 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7843122b-449a-4188-8755-b49147f6f8ed-tls-certs\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.006674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.006636 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-tmp-dir\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.006921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.006760 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-home\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.006921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.006806 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpkf\" (UniqueName: \"kubernetes.io/projected/7843122b-449a-4188-8755-b49147f6f8ed-kube-api-access-tbpkf\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.006921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.006902 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-model-cache\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.007079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.006938 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-dshm\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.007079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.007031 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-tmp-dir\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.007200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.007185 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-home\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.007374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.007348 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-model-cache\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.009357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.009332 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-dshm\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.009722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.009703 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7843122b-449a-4188-8755-b49147f6f8ed-tls-certs\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.069093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.069057 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpkf\" (UniqueName: \"kubernetes.io/projected/7843122b-449a-4188-8755-b49147f6f8ed-kube-api-access-tbpkf\") pod \"simulator-w6-547a38-kserve-6b9c556555-c2gsp\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.357547 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.357445 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:15.736396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:15.736315 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp"] Nov 03 16:24:15.738538 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:24:15.738512 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7843122b_449a_4188_8755_b49147f6f8ed.slice/crio-9e161db45ff83fc3b766d51907e85a23850a689e2ee5b1fb9dae397e977a3380 WatchSource:0}: Error finding container 9e161db45ff83fc3b766d51907e85a23850a689e2ee5b1fb9dae397e977a3380: Status 404 returned error can't find the container with id 9e161db45ff83fc3b766d51907e85a23850a689e2ee5b1fb9dae397e977a3380 Nov 03 16:24:16.060720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:16.060624 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:16.672511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:16.672473 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" event={"ID":"7843122b-449a-4188-8755-b49147f6f8ed","Type":"ContainerStarted","Data":"c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289"} Nov 03 16:24:16.672709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:16.672521 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" event={"ID":"7843122b-449a-4188-8755-b49147f6f8ed","Type":"ContainerStarted","Data":"9e161db45ff83fc3b766d51907e85a23850a689e2ee5b1fb9dae397e977a3380"} Nov 03 16:24:16.795851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:16.795796 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podStartSLOduration=2.795783063 podStartE2EDuration="2.795783063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:24:16.793866579 +0000 UTC m=+726.914919403" watchObservedRunningTime="2026-11-03 16:24:16.795783063 +0000 UTC m=+726.916835886" Nov 03 16:24:16.846858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:16.846827 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:17.357832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:17.357794 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:20.037297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:20.037235 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:20.846035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:20.846000 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:25.358190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:25.358153 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:25.370800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:25.370779 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:25.577564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:25.577534 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:25.736362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:25.736330 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:24:27.100468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.100430 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2"] Nov 03 16:24:27.104174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.104158 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.210574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.210544 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"default-dockercfg-q6dvs\"" Nov 03 16:24:27.210752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.210589 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"simulator-w1-bcd756-kserve-self-signed-certs\"" Nov 03 16:24:27.210752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.210590 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"kube-root-ca.crt\"" Nov 03 16:24:27.210901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.210888 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"openshift-service-ca.crt\"" Nov 03 16:24:27.213553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.213529 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2"] Nov 03 16:24:27.233651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.233618 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rls\" (UniqueName: \"kubernetes.io/projected/7cccbb33-75ea-43de-9340-41f1a7e45811-kube-api-access-f4rls\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.233781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.233750 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-home\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.233832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.233780 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-tmp-dir\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.233880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.233858 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cccbb33-75ea-43de-9340-41f1a7e45811-tls-certs\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.233918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.233904 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-dshm\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.234012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.234000 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-model-cache\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.246329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.243661 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:27.335274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335225 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-home\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.335274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335274 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-tmp-dir\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.335543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335303 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cccbb33-75ea-43de-9340-41f1a7e45811-tls-certs\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.335543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335365 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-dshm\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.335543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335422 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-model-cache\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.335543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335473 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rls\" (UniqueName: \"kubernetes.io/projected/7cccbb33-75ea-43de-9340-41f1a7e45811-kube-api-access-f4rls\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.335754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335692 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-tmp-dir\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.335809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335746 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-home\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.336020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.335995 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-model-cache\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.337837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.337816 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-dshm\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.338004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.337986 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cccbb33-75ea-43de-9340-41f1a7e45811-tls-certs\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.397196 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.397118 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rls\" (UniqueName: \"kubernetes.io/projected/7cccbb33-75ea-43de-9340-41f1a7e45811-kube-api-access-f4rls\") pod \"simulator-w1-bcd756-kserve-f56ddcc75-hspg2\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.417039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.417006 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:27.810569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:27.810539 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2"] Nov 03 16:24:27.812130 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:24:27.812106 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cccbb33_75ea_43de_9340_41f1a7e45811.slice/crio-bd673f4fe30b9212136b8211bbbd656ffc3d0b920c6d7c63f28979c095295384 WatchSource:0}: Error finding container bd673f4fe30b9212136b8211bbbd656ffc3d0b920c6d7c63f28979c095295384: Status 404 returned error can't find the container with id bd673f4fe30b9212136b8211bbbd656ffc3d0b920c6d7c63f28979c095295384 Nov 03 16:24:28.077938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:28.077818 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:28.411864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:28.411762 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:28.743460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:28.743428 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" event={"ID":"7cccbb33-75ea-43de-9340-41f1a7e45811","Type":"ContainerStarted","Data":"1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524"} Nov 03 16:24:28.743622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:28.743468 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" event={"ID":"7cccbb33-75ea-43de-9340-41f1a7e45811","Type":"ContainerStarted","Data":"bd673f4fe30b9212136b8211bbbd656ffc3d0b920c6d7c63f28979c095295384"} Nov 03 16:24:28.867368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:28.867314 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podStartSLOduration=1.8672998779999999 podStartE2EDuration="1.867299878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:24:28.866137865 +0000 UTC m=+738.987190691" watchObservedRunningTime="2026-11-03 16:24:28.867299878 +0000 UTC m=+738.988352701" Nov 03 16:24:29.417594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:29.417544 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:30.468264 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:30.468190 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:35.282864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:35.282825 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:37.418133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:37.418096 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:37.430610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:37.430581 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:37.803090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:37.803062 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:24:40.056982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:40.056926 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:40.304318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:40.304276 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:24:42.584884 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:24:42.584837 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:24:44.836088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:44.836006 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:49.655704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:49.655662 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:51.995300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:51.995263 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:54.462749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:54.462710 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:58.049569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:58.049532 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:24:59.265644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:24:59.265610 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:04.050590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:04.050535 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:08.860677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:08.860648 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:10.155184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:10.155148 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:10.346715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:10.346647 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:12.585292 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:25:12.585250 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:25:13.672134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:13.672098 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:15.792227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.792169 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg"] Nov 03 16:25:15.796724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.796701 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:15.894002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.893966 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"premium-w0-42da0f-kserve-self-signed-certs\"" Nov 03 16:25:15.904231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.904181 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg"] Nov 03 16:25:15.923388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.923360 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:15.943122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.943083 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-dshm\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:15.943360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.943130 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-home\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:15.943360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.943181 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q829\" (UniqueName: \"kubernetes.io/projected/c31c0b4b-ff16-4214-bb48-c3058890db7b-kube-api-access-4q829\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:15.943360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.943323 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-tmp-dir\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:15.943500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.943428 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-model-cache\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:15.943579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:15.943560 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31c0b4b-ff16-4214-bb48-c3058890db7b-tls-certs\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.044783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.044692 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q829\" (UniqueName: \"kubernetes.io/projected/c31c0b4b-ff16-4214-bb48-c3058890db7b-kube-api-access-4q829\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.044783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.044736 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-tmp-dir\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.045024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.044787 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-model-cache\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.045024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.044851 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31c0b4b-ff16-4214-bb48-c3058890db7b-tls-certs\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.045024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.044910 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-dshm\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.045024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.044935 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-home\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.045343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.045316 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-home\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.045488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.045345 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-tmp-dir\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.045488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.045447 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-model-cache\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.047422 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.047397 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-dshm\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.047701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.047684 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31c0b4b-ff16-4214-bb48-c3058890db7b-tls-certs\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.106463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.106427 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q829\" (UniqueName: \"kubernetes.io/projected/c31c0b4b-ff16-4214-bb48-c3058890db7b-kube-api-access-4q829\") pod \"premium-w0-42da0f-kserve-844bcbddff-rx2cg\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.109107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.109068 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:16.497892 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:25:16.497863 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc31c0b4b_ff16_4214_bb48_c3058890db7b.slice/crio-28d2232e77d9bbbe5a1a462f3b1aa031de06eb2377422244546a52c2ce9a183b WatchSource:0}: Error finding container 28d2232e77d9bbbe5a1a462f3b1aa031de06eb2377422244546a52c2ce9a183b: Status 404 returned error can't find the container with id 28d2232e77d9bbbe5a1a462f3b1aa031de06eb2377422244546a52c2ce9a183b Nov 03 16:25:16.500439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:16.500380 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg"] Nov 03 16:25:17.010888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:17.010851 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" event={"ID":"c31c0b4b-ff16-4214-bb48-c3058890db7b","Type":"ContainerStarted","Data":"6bfa5d7078db55ab34454f18674ae5135e3d6cfc44fa3d2c0066089a565fbfbf"} Nov 03 16:25:17.011329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:17.010901 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" event={"ID":"c31c0b4b-ff16-4214-bb48-c3058890db7b","Type":"ContainerStarted","Data":"28d2232e77d9bbbe5a1a462f3b1aa031de06eb2377422244546a52c2ce9a183b"} Nov 03 16:25:17.109426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:17.109392 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:17.136291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:17.136233 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podStartSLOduration=2.136186027 podStartE2EDuration="2.136186027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:25:17.13313559 +0000 UTC m=+787.254188412" watchObservedRunningTime="2026-11-03 16:25:17.136186027 +0000 UTC m=+787.257238850" Nov 03 16:25:18.460474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:18.460431 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:21.850295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:21.850263 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:25:23.236776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:23.236738 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:26.109308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:26.109269 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:26.121993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:26.121967 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:27.074101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:27.074074 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:25:27.981510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:27.981468 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:28.246302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:28.246182 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:28.444777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:28.444735 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:32.778002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:32.777965 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:37.558581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:37.558544 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:39.433480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:39.433397 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:42.362695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:42.362633 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:42.586307 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:25:42.586259 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:25:47.172948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:47.172903 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:50.795380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:50.795341 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:25:51.979890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:51.979851 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:55.904163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:55.904130 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:56.875651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:56.875612 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:58.187783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.187738 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht"] Nov 03 16:25:58.191618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.191594 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.249146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.249113 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht"] Nov 03 16:25:58.297129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.297087 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqkc\" (UniqueName: \"kubernetes.io/projected/d1b3fd26-7309-492e-a67f-6ad9106473e2-kube-api-access-ndqkc\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.297341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.297149 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.297341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.297263 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:25:58.297341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.297285 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:25:58.297341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.297318 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.297341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.297330 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:25:58.297562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.297382 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:25:58.398918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.398884 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.399089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.398948 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.399089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.399046 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqkc\" (UniqueName: \"kubernetes.io/projected/d1b3fd26-7309-492e-a67f-6ad9106473e2-kube-api-access-ndqkc\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.399932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.399907 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.401532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.401506 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.462877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.462813 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqkc\" (UniqueName: \"kubernetes.io/projected/d1b3fd26-7309-492e-a67f-6ad9106473e2-kube-api-access-ndqkc\") pod \"maas-api-revoke-sub-0ce6ee70-rhjht\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.504600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.504565 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:25:58.882513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:58.882487 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht"] Nov 03 16:25:58.885290 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:25:58.885265 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b3fd26_7309_492e_a67f_6ad9106473e2.slice/crio-f00c7ef1d0a1513de3c65f2363ccb5af89a5439eda2a3ec0fb431af4747ad82a WatchSource:0}: Error finding container f00c7ef1d0a1513de3c65f2363ccb5af89a5439eda2a3ec0fb431af4747ad82a: Status 404 returned error can't find the container with id f00c7ef1d0a1513de3c65f2363ccb5af89a5439eda2a3ec0fb431af4747ad82a Nov 03 16:25:59.236186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:25:59.236153 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" event={"ID":"d1b3fd26-7309-492e-a67f-6ad9106473e2","Type":"ContainerStarted","Data":"f00c7ef1d0a1513de3c65f2363ccb5af89a5439eda2a3ec0fb431af4747ad82a"} Nov 03 16:26:01.679185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:01.679145 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:04.268147 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:04.268103 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" event={"ID":"d1b3fd26-7309-492e-a67f-6ad9106473e2","Type":"ContainerStarted","Data":"5fa7acebd89368424d490b611be38120ccb0b7801f552ea253c9f2b369ef45ff"} Nov 03 16:26:04.392802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:04.392686 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" podStartSLOduration=1.50407292 podStartE2EDuration="6.392671167s" totalImagesPullingTime="4.888598247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:26:04.391001067 +0000 UTC m=+834.512053890" watchObservedRunningTime="2026-11-03 16:26:04.392671167 +0000 UTC m=+834.513723990" Nov 03 16:26:05.275629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:05.275588 5946 generic.go:356] "Generic (PLEG): container finished" podID="d1b3fd26-7309-492e-a67f-6ad9106473e2" containerID="5fa7acebd89368424d490b611be38120ccb0b7801f552ea253c9f2b369ef45ff" exitCode=0 Nov 03 16:26:05.276039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:05.275669 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" event={"ID":"d1b3fd26-7309-492e-a67f-6ad9106473e2","Type":"ContainerDied","Data":"5fa7acebd89368424d490b611be38120ccb0b7801f552ea253c9f2b369ef45ff"} Nov 03 16:26:06.429488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.429457 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:26:06.485924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.485891 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:06.587859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.587773 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-openshift-ca\") pod \"d1b3fd26-7309-492e-a67f-6ad9106473e2\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " Nov 03 16:26:06.587859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.587830 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-xks-ca\") pod \"d1b3fd26-7309-492e-a67f-6ad9106473e2\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " Nov 03 16:26:06.588084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.587913 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b3fd26-7309-492e-a67f-6ad9106473e2-kube-api-access-ndqkc\" (UniqueName: \"kubernetes.io/projected/d1b3fd26-7309-492e-a67f-6ad9106473e2-kube-api-access-ndqkc\") pod \"d1b3fd26-7309-492e-a67f-6ad9106473e2\" (UID: \"d1b3fd26-7309-492e-a67f-6ad9106473e2\") " Nov 03 16:26:06.588157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.588122 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-openshift-ca" pod "d1b3fd26-7309-492e-a67f-6ad9106473e2" (UID: "d1b3fd26-7309-492e-a67f-6ad9106473e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:06.590361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.590328 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-xks-ca" pod "d1b3fd26-7309-492e-a67f-6ad9106473e2" (UID: "d1b3fd26-7309-492e-a67f-6ad9106473e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:06.590483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.590377 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b3fd26-7309-492e-a67f-6ad9106473e2-kube-api-access-ndqkc" pod "d1b3fd26-7309-492e-a67f-6ad9106473e2" (UID: "d1b3fd26-7309-492e-a67f-6ad9106473e2"). InnerVolumeSpecName "kube-api-access-ndqkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:06.688919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.688881 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:06.688919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.688913 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b3fd26-7309-492e-a67f-6ad9106473e2-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:06.688919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:06.688925 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndqkc\" (UniqueName: \"kubernetes.io/projected/d1b3fd26-7309-492e-a67f-6ad9106473e2-kube-api-access-ndqkc\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:07.287201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:07.287166 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" event={"ID":"d1b3fd26-7309-492e-a67f-6ad9106473e2","Type":"ContainerDied","Data":"f00c7ef1d0a1513de3c65f2363ccb5af89a5439eda2a3ec0fb431af4747ad82a"} Nov 03 16:26:07.287417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:07.287197 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht" Nov 03 16:26:07.287417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:07.287236 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f00c7ef1d0a1513de3c65f2363ccb5af89a5439eda2a3ec0fb431af4747ad82a" Nov 03 16:26:08.837333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.837299 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r"] Nov 03 16:26:08.837758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.837744 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d1b3fd26-7309-492e-a67f-6ad9106473e2" containerName="revoke-keys" Nov 03 16:26:08.837803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.837759 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b3fd26-7309-492e-a67f-6ad9106473e2" containerName="revoke-keys" Nov 03 16:26:08.837867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.837858 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b3fd26-7309-492e-a67f-6ad9106473e2" containerName="revoke-keys" Nov 03 16:26:08.841340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.841296 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:08.901436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.901402 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r"] Nov 03 16:26:08.915392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.915362 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tc79\" (UniqueName: \"kubernetes.io/projected/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-kube-api-access-4tc79\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:08.915548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.915489 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:08.915589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.915554 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:08.949277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.949240 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:26:08.949424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.949349 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:26:08.949424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.949361 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:26:08.949424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:08.949360 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:26:09.016690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.016660 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:09.016847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.016730 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tc79\" (UniqueName: \"kubernetes.io/projected/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-kube-api-access-4tc79\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:09.016847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.016788 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:09.017441 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.017419 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:09.019305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.019281 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:09.079549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.079516 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tc79\" (UniqueName: \"kubernetes.io/projected/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-kube-api-access-4tc79\") pod \"maas-api-revoke-sub-3fd3ce74-9dh7r\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:09.155091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.155012 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:09.541015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:09.540990 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r"] Nov 03 16:26:09.542603 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:09.542577 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7b0715_d949_4b9f_afb1_ec7499b9b6d5.slice/crio-563f0061f45cc08a362927de7061654ace7fdeaa62f7c45bbb5e6a0ba73365c4 WatchSource:0}: Error finding container 563f0061f45cc08a362927de7061654ace7fdeaa62f7c45bbb5e6a0ba73365c4: Status 404 returned error can't find the container with id 563f0061f45cc08a362927de7061654ace7fdeaa62f7c45bbb5e6a0ba73365c4 Nov 03 16:26:10.305711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:10.305674 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" event={"ID":"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5","Type":"ContainerStarted","Data":"a59735cb94c1349399aa85edc638b61215f003eafbc87c9365fb2a90ecb0ec21"} Nov 03 16:26:10.306102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:10.305769 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" event={"ID":"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5","Type":"ContainerStarted","Data":"563f0061f45cc08a362927de7061654ace7fdeaa62f7c45bbb5e6a0ba73365c4"} Nov 03 16:26:10.432824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:10.432762 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" podStartSLOduration=2.432748388 podStartE2EDuration="2.432748388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:26:10.431491939 +0000 UTC m=+840.552544763" watchObservedRunningTime="2026-11-03 16:26:10.432748388 +0000 UTC m=+840.553801234" Nov 03 16:26:11.281026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:11.280994 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:11.313134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:11.313096 5946 generic.go:356] "Generic (PLEG): container finished" podID="fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" containerID="a59735cb94c1349399aa85edc638b61215f003eafbc87c9365fb2a90ecb0ec21" exitCode=0 Nov 03 16:26:11.313134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:11.313121 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" event={"ID":"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5","Type":"ContainerDied","Data":"a59735cb94c1349399aa85edc638b61215f003eafbc87c9365fb2a90ecb0ec21"} Nov 03 16:26:12.473810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.473786 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:12.555896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.555859 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-kube-api-access-4tc79\" (UniqueName: \"kubernetes.io/projected/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-kube-api-access-4tc79\") pod \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " Nov 03 16:26:12.556058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.555912 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-xks-ca\") pod \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " Nov 03 16:26:12.556100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.556072 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-openshift-ca\") pod \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\" (UID: \"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5\") " Nov 03 16:26:12.556449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.556426 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-openshift-ca" pod "fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" (UID: "fa7b0715-d949-4b9f-afb1-ec7499b9b6d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:12.558172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.558143 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-xks-ca" pod "fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" (UID: "fa7b0715-d949-4b9f-afb1-ec7499b9b6d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:12.558311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.558235 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-kube-api-access-4tc79" pod "fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" (UID: "fa7b0715-d949-4b9f-afb1-ec7499b9b6d5"). InnerVolumeSpecName "kube-api-access-4tc79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:12.586244 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:26:12.586140 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:26:12.657591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.657562 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:12.657591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.657587 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tc79\" (UniqueName: \"kubernetes.io/projected/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-kube-api-access-4tc79\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:12.657776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:12.657600 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:13.326863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:13.326833 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" Nov 03 16:26:13.327041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:13.326865 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r" event={"ID":"fa7b0715-d949-4b9f-afb1-ec7499b9b6d5","Type":"ContainerDied","Data":"563f0061f45cc08a362927de7061654ace7fdeaa62f7c45bbb5e6a0ba73365c4"} Nov 03 16:26:13.327041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:13.326921 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563f0061f45cc08a362927de7061654ace7fdeaa62f7c45bbb5e6a0ba73365c4" Nov 03 16:26:15.974322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:15.974290 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:17.229434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:17.229402 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:18.237699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:18.237663 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:18.754690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:18.754643 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv"] Nov 03 16:26:18.754902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:18.754869 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" containerID="cri-o://0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142" gracePeriod=30 Nov 03 16:26:19.373294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:19.373260 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:19.564277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:19.564243 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:20.759715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:20.759687 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:24.018776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.018713 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:26:24.076985 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.076949 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-certs\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.076985 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.076991 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-envoy\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077015 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-credential-socket\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077047 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-kube-api-access-xlck5\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-kube-api-access-xlck5\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077077 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-podinfo\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077258 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-certs" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:26:24.077412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077303 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-socket\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077357 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-token\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-token\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077365 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-credential-socket" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:26:24.077412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077400 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-data\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077431 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83eaba29-3d02-41ab-bad7-c4d076ca6215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83eaba29-3d02-41ab-bad7-c4d076ca6215-istiod-ca-cert\") pod \"83eaba29-3d02-41ab-bad7-c4d076ca6215\" (UID: \"83eaba29-3d02-41ab-bad7-c4d076ca6215\") " Nov 03 16:26:24.077657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077562 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-socket" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:26:24.077754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077676 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-data" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:26:24.077921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.077896 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eaba29-3d02-41ab-bad7-c4d076ca6215-istiod-ca-cert" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:24.078693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.078619 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.078693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.078669 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.078693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.078687 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.078900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.078702 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.078900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.078716 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83eaba29-3d02-41ab-bad7-c4d076ca6215-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.079757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.079737 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-kube-api-access-xlck5" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "kube-api-access-xlck5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:24.079957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.079931 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-envoy" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:26:24.080051 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.079995 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-token" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:24.080174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.080161 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-podinfo" pod "83eaba29-3d02-41ab-bad7-c4d076ca6215" (UID: "83eaba29-3d02-41ab-bad7-c4d076ca6215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:26:24.180111 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.180079 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.180111 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.180104 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.180354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.180115 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlck5\" (UniqueName: \"kubernetes.io/projected/83eaba29-3d02-41ab-bad7-c4d076ca6215-kube-api-access-xlck5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.180354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.180128 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83eaba29-3d02-41ab-bad7-c4d076ca6215-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:24.297519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.297434 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4"] Nov 03 16:26:24.297892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.297878 5946 cpu_manager.go:414] "containerMap: removing container" podUID="fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" containerName="revoke-keys" Nov 03 16:26:24.297936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.297893 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" containerName="revoke-keys" Nov 03 16:26:24.297936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.297912 5946 cpu_manager.go:414] "containerMap: removing container" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" Nov 03 16:26:24.297936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.297918 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" Nov 03 16:26:24.298032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.298000 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" containerName="revoke-keys" Nov 03 16:26:24.298032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.298010 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerName="istio-proxy" Nov 03 16:26:24.301568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.301549 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.358834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.358804 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:26:24.358834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.358820 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:26:24.359015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.358940 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:26:24.364866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.364838 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4"] Nov 03 16:26:24.379154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.379122 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2"] Nov 03 16:26:24.382221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.382171 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z49c\" (UniqueName: \"kubernetes.io/projected/5434235a-5555-4b2a-ac85-87c33495e2b4-kube-api-access-8z49c\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.382369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.382281 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.382369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.382355 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.382789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.382771 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.389313 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.389284 5946 generic.go:356] "Generic (PLEG): container finished" podID="83eaba29-3d02-41ab-bad7-c4d076ca6215" containerID="0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142" exitCode=0 Nov 03 16:26:24.389457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.389354 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" Nov 03 16:26:24.389457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.389374 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" event={"ID":"83eaba29-3d02-41ab-bad7-c4d076ca6215","Type":"ContainerDied","Data":"0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142"} Nov 03 16:26:24.389457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.389426 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv" event={"ID":"83eaba29-3d02-41ab-bad7-c4d076ca6215","Type":"ContainerDied","Data":"657d4fcfd72b8da0e3d3d7ee5613dde51f576abc3b7e8181dee4f013de07418d"} Nov 03 16:26:24.389457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.389454 5946 scope.go:122] "RemoveContainer" containerID="0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142" Nov 03 16:26:24.401326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.401309 5946 scope.go:122] "RemoveContainer" containerID="0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142" Nov 03 16:26:24.401613 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:26:24.401595 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142\": container with ID starting with 0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142 not found: ID does not exist" containerID="0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142" Nov 03 16:26:24.401657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.401621 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142"} err="failed to get container status \"0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142\": rpc error: code = NotFound desc = could not find container \"0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142\": container with ID starting with 0bf28850a29ddc97c01b8d9d9d27de26afca6499d2235db65a3a60431e4f0142 not found: ID does not exist" Nov 03 16:26:24.446854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.446816 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2"] Nov 03 16:26:24.458691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.458665 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:26:24.483284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.483239 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z49c\" (UniqueName: \"kubernetes.io/projected/5434235a-5555-4b2a-ac85-87c33495e2b4-kube-api-access-8z49c\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.483465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.483297 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.483465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.483327 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.483465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.483388 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.483609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.483552 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.483669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.483646 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87sq6\" (UniqueName: \"kubernetes.io/projected/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-kube-api-access-87sq6\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.484048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.484027 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.485955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.485936 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.546473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.546447 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z49c\" (UniqueName: \"kubernetes.io/projected/5434235a-5555-4b2a-ac85-87c33495e2b4-kube-api-access-8z49c\") pod \"maas-api-revoke-sub-0c436991-5djd4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.585038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.584940 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.585255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.585048 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.585255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.585152 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sq6\" (UniqueName: \"kubernetes.io/projected/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-kube-api-access-87sq6\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.585809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.585774 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.587473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.587456 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.615291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.615262 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:24.645697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.645671 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sq6\" (UniqueName: \"kubernetes.io/projected/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-kube-api-access-87sq6\") pod \"maas-api-revoke-sub-48df6cf2-pqhb2\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.657934 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.657877 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv"] Nov 03 16:26:24.664237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.664190 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-scnzv"] Nov 03 16:26:24.699262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.698995 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:24.998478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:24.998450 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4"] Nov 03 16:26:24.999870 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:24.999841 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5434235a_5555_4b2a_ac85_87c33495e2b4.slice/crio-424866d6b6a96587d4b8bd9a6d348f2c71cfb0882e1b278ab6bad1eed65f58c0 WatchSource:0}: Error finding container 424866d6b6a96587d4b8bd9a6d348f2c71cfb0882e1b278ab6bad1eed65f58c0: Status 404 returned error can't find the container with id 424866d6b6a96587d4b8bd9a6d348f2c71cfb0882e1b278ab6bad1eed65f58c0 Nov 03 16:26:25.112978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:25.112954 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2"] Nov 03 16:26:25.114933 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:25.114906 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa18c3f5_a2f1_4f13_97c7_c5a79e14b38c.slice/crio-fdde25945e703d3dbec0d5b2d2bcc617a9c2c1f58cfbd747771c8677d6b5f623 WatchSource:0}: Error finding container fdde25945e703d3dbec0d5b2d2bcc617a9c2c1f58cfbd747771c8677d6b5f623: Status 404 returned error can't find the container with id fdde25945e703d3dbec0d5b2d2bcc617a9c2c1f58cfbd747771c8677d6b5f623 Nov 03 16:26:25.396256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:25.396198 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" event={"ID":"5434235a-5555-4b2a-ac85-87c33495e2b4","Type":"ContainerStarted","Data":"550bc001c6694ae44e1e4adc17ac4abd00bc935f8358ae121a6dc3510d592ec8"} Nov 03 16:26:25.396450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:25.396289 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" event={"ID":"5434235a-5555-4b2a-ac85-87c33495e2b4","Type":"ContainerStarted","Data":"424866d6b6a96587d4b8bd9a6d348f2c71cfb0882e1b278ab6bad1eed65f58c0"} Nov 03 16:26:25.397582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:25.397555 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" event={"ID":"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c","Type":"ContainerStarted","Data":"fdde25945e703d3dbec0d5b2d2bcc617a9c2c1f58cfbd747771c8677d6b5f623"} Nov 03 16:26:25.517365 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:25.517315 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" podStartSLOduration=1.517300143 podStartE2EDuration="1.517300143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:26:25.515549577 +0000 UTC m=+855.636602474" watchObservedRunningTime="2026-11-03 16:26:25.517300143 +0000 UTC m=+855.638352966" Nov 03 16:26:25.565946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:25.565909 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:26.003864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:26.003832 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83eaba29-3d02-41ab-bad7-c4d076ca6215" path="/var/lib/kubelet/pods/83eaba29-3d02-41ab-bad7-c4d076ca6215/volumes" Nov 03 16:26:26.411876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:26.411844 5946 generic.go:356] "Generic (PLEG): container finished" podID="5434235a-5555-4b2a-ac85-87c33495e2b4" containerID="550bc001c6694ae44e1e4adc17ac4abd00bc935f8358ae121a6dc3510d592ec8" exitCode=0 Nov 03 16:26:26.412390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:26.411933 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" event={"ID":"5434235a-5555-4b2a-ac85-87c33495e2b4","Type":"ContainerDied","Data":"550bc001c6694ae44e1e4adc17ac4abd00bc935f8358ae121a6dc3510d592ec8"} Nov 03 16:26:26.413550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:26.413529 5946 generic.go:356] "Generic (PLEG): container finished" podID="aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" containerID="3c33f2e0760281055a5bfc2d71bf56c061be38f521d8f8989fa053016876866d" exitCode=0 Nov 03 16:26:26.413655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:26.413590 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" event={"ID":"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c","Type":"ContainerDied","Data":"3c33f2e0760281055a5bfc2d71bf56c061be38f521d8f8989fa053016876866d"} Nov 03 16:26:26.683797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:26.683710 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:26:27.562367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.562341 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:27.597010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.596986 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:27.614530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.614503 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-openshift-ca\") pod \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " Nov 03 16:26:27.614694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.614584 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-kube-api-access-87sq6\" (UniqueName: \"kubernetes.io/projected/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-kube-api-access-87sq6\") pod \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " Nov 03 16:26:27.614694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.614632 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-xks-ca\") pod \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\" (UID: \"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c\") " Nov 03 16:26:27.614879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.614854 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-openshift-ca" pod "aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" (UID: "aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:27.616796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.616769 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-xks-ca" pod "aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" (UID: "aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:27.616881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.616813 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-kube-api-access-87sq6" pod "aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" (UID: "aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c"). InnerVolumeSpecName "kube-api-access-87sq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:27.716192 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.716096 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-openshift-ca\") pod \"5434235a-5555-4b2a-ac85-87c33495e2b4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " Nov 03 16:26:27.716192 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.716151 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-xks-ca\") pod \"5434235a-5555-4b2a-ac85-87c33495e2b4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " Nov 03 16:26:27.716408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.716264 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5434235a-5555-4b2a-ac85-87c33495e2b4-kube-api-access-8z49c\" (UniqueName: \"kubernetes.io/projected/5434235a-5555-4b2a-ac85-87c33495e2b4-kube-api-access-8z49c\") pod \"5434235a-5555-4b2a-ac85-87c33495e2b4\" (UID: \"5434235a-5555-4b2a-ac85-87c33495e2b4\") " Nov 03 16:26:27.716586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.716550 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-openshift-ca" pod "5434235a-5555-4b2a-ac85-87c33495e2b4" (UID: "5434235a-5555-4b2a-ac85-87c33495e2b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:27.718481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.718450 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:27.718481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.718483 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87sq6\" (UniqueName: \"kubernetes.io/projected/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-kube-api-access-87sq6\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:27.718645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.718499 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:27.718645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.718514 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:27.718645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.718477 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5434235a-5555-4b2a-ac85-87c33495e2b4-kube-api-access-8z49c" pod "5434235a-5555-4b2a-ac85-87c33495e2b4" (UID: "5434235a-5555-4b2a-ac85-87c33495e2b4"). InnerVolumeSpecName "kube-api-access-8z49c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:27.718645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.718537 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-xks-ca" pod "5434235a-5555-4b2a-ac85-87c33495e2b4" (UID: "5434235a-5555-4b2a-ac85-87c33495e2b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:27.819443 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.819405 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z49c\" (UniqueName: \"kubernetes.io/projected/5434235a-5555-4b2a-ac85-87c33495e2b4-kube-api-access-8z49c\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:27.819443 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:27.819438 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5434235a-5555-4b2a-ac85-87c33495e2b4-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:28.426166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:28.426079 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" Nov 03 16:26:28.426166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:28.426104 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4" event={"ID":"5434235a-5555-4b2a-ac85-87c33495e2b4","Type":"ContainerDied","Data":"424866d6b6a96587d4b8bd9a6d348f2c71cfb0882e1b278ab6bad1eed65f58c0"} Nov 03 16:26:28.426166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:28.426158 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="424866d6b6a96587d4b8bd9a6d348f2c71cfb0882e1b278ab6bad1eed65f58c0" Nov 03 16:26:28.427685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:28.427667 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" Nov 03 16:26:28.427791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:28.427707 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2" event={"ID":"aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c","Type":"ContainerDied","Data":"fdde25945e703d3dbec0d5b2d2bcc617a9c2c1f58cfbd747771c8677d6b5f623"} Nov 03 16:26:28.427791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:28.427745 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdde25945e703d3dbec0d5b2d2bcc617a9c2c1f58cfbd747771c8677d6b5f623" Nov 03 16:26:28.786154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:28.786115 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:29.099939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:29.099823 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:29.288566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:29.288512 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:30.370860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:30.370830 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:35.170022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:35.169974 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:39.962036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:39.961989 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:42.226071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.226036 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv"] Nov 03 16:26:42.226488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.226475 5946 cpu_manager.go:414] "containerMap: removing container" podUID="5434235a-5555-4b2a-ac85-87c33495e2b4" containerName="revoke-keys" Nov 03 16:26:42.226531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.226489 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5434235a-5555-4b2a-ac85-87c33495e2b4" containerName="revoke-keys" Nov 03 16:26:42.226531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.226511 5946 cpu_manager.go:414] "containerMap: removing container" podUID="aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" containerName="revoke-keys" Nov 03 16:26:42.226531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.226516 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" containerName="revoke-keys" Nov 03 16:26:42.226629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.226591 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="5434235a-5555-4b2a-ac85-87c33495e2b4" containerName="revoke-keys" Nov 03 16:26:42.226629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.226610 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" containerName="revoke-keys" Nov 03 16:26:42.233970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.233949 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.289729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.289709 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6ae690-openshift-default-dockercfg-6s4h2\"" Nov 03 16:26:42.344793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.344753 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv"] Nov 03 16:26:42.372799 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.372773 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.372958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.372808 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.372958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.372909 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.372958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.372951 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.373086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.373049 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.373122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.373085 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.373122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.373107 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmmzc\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-kube-api-access-fmmzc\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.373272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.373250 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.373326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.373281 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474139 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474184 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474251 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474278 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474294 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmmzc\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-kube-api-access-fmmzc\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474336 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474360 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474394 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474425 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474615 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474690 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474768 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.474967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.474946 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.475062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.475043 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.476752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.476709 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.477020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.477002 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.535824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.535791 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.535960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.535863 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmmzc\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-kube-api-access-fmmzc\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.546757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.546734 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:42.586244 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:26:42.586182 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:26:42.983361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.983339 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv"] Nov 03 16:26:42.985295 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:42.985267 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75d8da1_b8d2_449f_bbb6_fa0dd60eff87.slice/crio-aa3c0a85b7842f0001d30694ec59db058bfdf77ae5c8d8279a838f45ee8debd7 WatchSource:0}: Error finding container aa3c0a85b7842f0001d30694ec59db058bfdf77ae5c8d8279a838f45ee8debd7: Status 404 returned error can't find the container with id aa3c0a85b7842f0001d30694ec59db058bfdf77ae5c8d8279a838f45ee8debd7 Nov 03 16:26:42.987628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.987596 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:26:42.987742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.987675 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:26:42.987742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:42.987720 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:26:43.520700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:43.520659 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" event={"ID":"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87","Type":"ContainerStarted","Data":"10ca7d68127047967c54968865d4aa337aeb39658ee1d10198f85f34d8c76867"} Nov 03 16:26:43.521360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:43.521335 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" event={"ID":"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87","Type":"ContainerStarted","Data":"aa3c0a85b7842f0001d30694ec59db058bfdf77ae5c8d8279a838f45ee8debd7"} Nov 03 16:26:43.547041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:43.547009 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:43.707930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:43.707880 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podStartSLOduration=1.707866644 podStartE2EDuration="1.707866644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:26:43.707041681 +0000 UTC m=+873.828094503" watchObservedRunningTime="2026-11-03 16:26:43.707866644 +0000 UTC m=+873.828919467" Nov 03 16:26:44.549407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.549374 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:26:44.549835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.549437 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 16:26:44.739242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.738944 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r"] Nov 03 16:26:44.743612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.743583 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:44.798951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.798918 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:26:44.805963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.805869 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:44.849047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.849001 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:26:44.849047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.849023 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:26:44.849047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.849001 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:26:44.851588 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.851563 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r"] Nov 03 16:26:44.903944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.903914 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrcrb\" (UniqueName: \"kubernetes.io/projected/ef3fcbb1-7a72-4bba-a064-423790337ce0-kube-api-access-xrcrb\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:44.904130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.903973 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:44.904130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:44.904054 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.005730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.005690 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrcrb\" (UniqueName: \"kubernetes.io/projected/ef3fcbb1-7a72-4bba-a064-423790337ce0-kube-api-access-xrcrb\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.005920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.005747 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.005920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.005767 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.006498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.006471 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.008361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.008338 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.067597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.067524 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrcrb\" (UniqueName: \"kubernetes.io/projected/ef3fcbb1-7a72-4bba-a064-423790337ce0-kube-api-access-xrcrb\") pod \"maas-api-revoke-sub-da4b021b-26p2r\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.358590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.358493 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:45.547310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.547266 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:26:45.547544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.547347 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 16:26:45.749860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:45.749838 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r"] Nov 03 16:26:45.751520 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:45.751494 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3fcbb1_7a72_4bba_a064_423790337ce0.slice/crio-af585d2fb2e14607a954c9dd3b8ab390a1ae9a25b336a2a2fe892184c5467f43 WatchSource:0}: Error finding container af585d2fb2e14607a954c9dd3b8ab390a1ae9a25b336a2a2fe892184c5467f43: Status 404 returned error can't find the container with id af585d2fb2e14607a954c9dd3b8ab390a1ae9a25b336a2a2fe892184c5467f43 Nov 03 16:26:46.548366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.548333 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:26:46.548542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.548384 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 16:26:46.558245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.558189 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" event={"ID":"ef3fcbb1-7a72-4bba-a064-423790337ce0","Type":"ContainerStarted","Data":"c72acec27ef36487e52c8f5aa915274341bd2d7d5cd8f99974b621de8ac15156"} Nov 03 16:26:46.558382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.558274 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" event={"ID":"ef3fcbb1-7a72-4bba-a064-423790337ce0","Type":"ContainerStarted","Data":"af585d2fb2e14607a954c9dd3b8ab390a1ae9a25b336a2a2fe892184c5467f43"} Nov 03 16:26:46.661709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.661674 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw"] Nov 03 16:26:46.665412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.665396 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.703176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.703144 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw"] Nov 03 16:26:46.712767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.712709 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" podStartSLOduration=2.712696012 podStartE2EDuration="2.712696012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:26:46.70136481 +0000 UTC m=+876.822417631" watchObservedRunningTime="2026-11-03 16:26:46.712696012 +0000 UTC m=+876.833748834" Nov 03 16:26:46.847917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.847838 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgbqp\" (UniqueName: \"kubernetes.io/projected/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-kube-api-access-pgbqp\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.848353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.847959 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.848353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.847990 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.949502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.949462 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbqp\" (UniqueName: \"kubernetes.io/projected/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-kube-api-access-pgbqp\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.949692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.949535 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.949692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.949558 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.950229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.950183 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:46.952009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:46.951992 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:47.012007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.011978 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbqp\" (UniqueName: \"kubernetes.io/projected/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-kube-api-access-pgbqp\") pod \"maas-api-revoke-sub-a5a7c4d6-4j2bw\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:47.279367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.279334 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:47.547997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.547890 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:26:47.547997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.547948 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 16:26:47.567122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.567088 5946 generic.go:356] "Generic (PLEG): container finished" podID="ef3fcbb1-7a72-4bba-a064-423790337ce0" containerID="c72acec27ef36487e52c8f5aa915274341bd2d7d5cd8f99974b621de8ac15156" exitCode=0 Nov 03 16:26:47.567320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.567166 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" event={"ID":"ef3fcbb1-7a72-4bba-a064-423790337ce0","Type":"ContainerDied","Data":"c72acec27ef36487e52c8f5aa915274341bd2d7d5cd8f99974b621de8ac15156"} Nov 03 16:26:47.685326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.685293 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw"] Nov 03 16:26:47.685700 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:47.685677 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ad70a9_6ab8_4eff_a50b_8dc10155b33a.slice/crio-dde57287d5301ea4e24a8bd534ebe5ac7bdc5f1a97ecbff6940753ae97dae9d5 WatchSource:0}: Error finding container dde57287d5301ea4e24a8bd534ebe5ac7bdc5f1a97ecbff6940753ae97dae9d5: Status 404 returned error can't find the container with id dde57287d5301ea4e24a8bd534ebe5ac7bdc5f1a97ecbff6940753ae97dae9d5 Nov 03 16:26:47.850685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.850602 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx"] Nov 03 16:26:47.855662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.855639 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:47.917234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.916153 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx"] Nov 03 16:26:47.959842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.959807 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjglf\" (UniqueName: \"kubernetes.io/projected/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-kube-api-access-cjglf\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:47.960037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.959886 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:47.960177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:47.960044 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.061071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.061033 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.061282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.061116 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjglf\" (UniqueName: \"kubernetes.io/projected/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-kube-api-access-cjglf\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.061282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.061229 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.061886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.061860 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.064240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.063603 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.137381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.137318 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjglf\" (UniqueName: \"kubernetes.io/projected/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-kube-api-access-cjglf\") pod \"maas-api-revoke-sub-fb18831b-wxqzx\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.168437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.168408 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:48.547619 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.547582 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:26:48.547802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.547643 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 16:26:48.574182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.574128 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" event={"ID":"22ad70a9-6ab8-4eff-a50b-8dc10155b33a","Type":"ContainerStarted","Data":"1e90c62fd222a9f9ff5a684d1f6674f9c37a5d1d504dc9131aadd01943280259"} Nov 03 16:26:48.574398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.574200 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" event={"ID":"22ad70a9-6ab8-4eff-a50b-8dc10155b33a","Type":"ContainerStarted","Data":"dde57287d5301ea4e24a8bd534ebe5ac7bdc5f1a97ecbff6940753ae97dae9d5"} Nov 03 16:26:48.577368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.577339 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx"] Nov 03 16:26:48.580641 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:48.580613 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7b15ab_7a30_45df_bb25_6f8ed973b0a0.slice/crio-df1c46cd768f2c0dccccf55095656e0b162c04c50dea0890be35321a4234b293 WatchSource:0}: Error finding container df1c46cd768f2c0dccccf55095656e0b162c04c50dea0890be35321a4234b293: Status 404 returned error can't find the container with id df1c46cd768f2c0dccccf55095656e0b162c04c50dea0890be35321a4234b293 Nov 03 16:26:48.701967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.701859 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" podStartSLOduration=2.7018380239999997 podStartE2EDuration="2.701838024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:26:48.699123341 +0000 UTC m=+878.820176166" watchObservedRunningTime="2026-11-03 16:26:48.701838024 +0000 UTC m=+878.822890848" Nov 03 16:26:48.764015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.763992 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:48.870847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.870813 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-xks-ca\") pod \"ef3fcbb1-7a72-4bba-a064-423790337ce0\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " Nov 03 16:26:48.871403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.870895 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-openshift-ca\") pod \"ef3fcbb1-7a72-4bba-a064-423790337ce0\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " Nov 03 16:26:48.871403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.871002 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3fcbb1-7a72-4bba-a064-423790337ce0-kube-api-access-xrcrb\" (UniqueName: \"kubernetes.io/projected/ef3fcbb1-7a72-4bba-a064-423790337ce0-kube-api-access-xrcrb\") pod \"ef3fcbb1-7a72-4bba-a064-423790337ce0\" (UID: \"ef3fcbb1-7a72-4bba-a064-423790337ce0\") " Nov 03 16:26:48.871480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.871392 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-openshift-ca" pod "ef3fcbb1-7a72-4bba-a064-423790337ce0" (UID: "ef3fcbb1-7a72-4bba-a064-423790337ce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:48.873482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.873447 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-xks-ca" pod "ef3fcbb1-7a72-4bba-a064-423790337ce0" (UID: "ef3fcbb1-7a72-4bba-a064-423790337ce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:48.873982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.873957 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3fcbb1-7a72-4bba-a064-423790337ce0-kube-api-access-xrcrb" pod "ef3fcbb1-7a72-4bba-a064-423790337ce0" (UID: "ef3fcbb1-7a72-4bba-a064-423790337ce0"). InnerVolumeSpecName "kube-api-access-xrcrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:48.972846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.972799 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:48.972846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.972838 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrcrb\" (UniqueName: \"kubernetes.io/projected/ef3fcbb1-7a72-4bba-a064-423790337ce0-kube-api-access-xrcrb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:48.972846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:48.972848 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3fcbb1-7a72-4bba-a064-423790337ce0-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:49.581935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.581897 5946 generic.go:356] "Generic (PLEG): container finished" podID="22ad70a9-6ab8-4eff-a50b-8dc10155b33a" containerID="1e90c62fd222a9f9ff5a684d1f6674f9c37a5d1d504dc9131aadd01943280259" exitCode=0 Nov 03 16:26:49.582110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.581984 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" event={"ID":"22ad70a9-6ab8-4eff-a50b-8dc10155b33a","Type":"ContainerDied","Data":"1e90c62fd222a9f9ff5a684d1f6674f9c37a5d1d504dc9131aadd01943280259"} Nov 03 16:26:49.583851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.583826 5946 generic.go:356] "Generic (PLEG): container finished" podID="db7b15ab-7a30-45df-bb25-6f8ed973b0a0" containerID="9b888124d0d6d4bf661229281a1662cd2474938df2e2e0cce67e3981fd2b9951" exitCode=0 Nov 03 16:26:49.583966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.583914 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" event={"ID":"db7b15ab-7a30-45df-bb25-6f8ed973b0a0","Type":"ContainerDied","Data":"9b888124d0d6d4bf661229281a1662cd2474938df2e2e0cce67e3981fd2b9951"} Nov 03 16:26:49.583966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.583960 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" event={"ID":"db7b15ab-7a30-45df-bb25-6f8ed973b0a0","Type":"ContainerStarted","Data":"df1c46cd768f2c0dccccf55095656e0b162c04c50dea0890be35321a4234b293"} Nov 03 16:26:49.585499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.585476 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" event={"ID":"ef3fcbb1-7a72-4bba-a064-423790337ce0","Type":"ContainerDied","Data":"af585d2fb2e14607a954c9dd3b8ab390a1ae9a25b336a2a2fe892184c5467f43"} Nov 03 16:26:49.585601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.585513 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r" Nov 03 16:26:49.585601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.585522 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af585d2fb2e14607a954c9dd3b8ab390a1ae9a25b336a2a2fe892184c5467f43" Nov 03 16:26:49.646262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.646235 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:49.646581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.646557 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:49.647458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.647437 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:26:49.772237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:49.771296 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:50.737062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.737033 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:50.765667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.765642 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:50.792085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792057 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-openshift-ca\") pod \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " Nov 03 16:26:50.792244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792141 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-xks-ca\") pod \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " Nov 03 16:26:50.792244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792180 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-openshift-ca\") pod \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " Nov 03 16:26:50.792244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792236 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-kube-api-access-cjglf\" (UniqueName: \"kubernetes.io/projected/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-kube-api-access-cjglf\") pod \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\" (UID: \"db7b15ab-7a30-45df-bb25-6f8ed973b0a0\") " Nov 03 16:26:50.792407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792274 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-kube-api-access-pgbqp\" (UniqueName: \"kubernetes.io/projected/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-kube-api-access-pgbqp\") pod \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " Nov 03 16:26:50.792407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792313 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-xks-ca\") pod \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\" (UID: \"22ad70a9-6ab8-4eff-a50b-8dc10155b33a\") " Nov 03 16:26:50.792652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792620 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-openshift-ca" pod "22ad70a9-6ab8-4eff-a50b-8dc10155b33a" (UID: "22ad70a9-6ab8-4eff-a50b-8dc10155b33a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:50.792781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792706 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-openshift-ca" pod "db7b15ab-7a30-45df-bb25-6f8ed973b0a0" (UID: "db7b15ab-7a30-45df-bb25-6f8ed973b0a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:50.792781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.792763 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:50.794881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.794847 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-xks-ca" pod "22ad70a9-6ab8-4eff-a50b-8dc10155b33a" (UID: "22ad70a9-6ab8-4eff-a50b-8dc10155b33a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:50.794881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.794848 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-kube-api-access-pgbqp" pod "22ad70a9-6ab8-4eff-a50b-8dc10155b33a" (UID: "22ad70a9-6ab8-4eff-a50b-8dc10155b33a"). InnerVolumeSpecName "kube-api-access-pgbqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:50.794881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.794871 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-kube-api-access-cjglf" pod "db7b15ab-7a30-45df-bb25-6f8ed973b0a0" (UID: "db7b15ab-7a30-45df-bb25-6f8ed973b0a0"). InnerVolumeSpecName "kube-api-access-cjglf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:50.795128 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.795110 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-xks-ca" pod "db7b15ab-7a30-45df-bb25-6f8ed973b0a0" (UID: "db7b15ab-7a30-45df-bb25-6f8ed973b0a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:50.894122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.894019 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:50.894122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.894054 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:50.894122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.894070 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjglf\" (UniqueName: \"kubernetes.io/projected/db7b15ab-7a30-45df-bb25-6f8ed973b0a0-kube-api-access-cjglf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:50.894122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.894082 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgbqp\" (UniqueName: \"kubernetes.io/projected/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-kube-api-access-pgbqp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:50.894122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:50.894096 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ad70a9-6ab8-4eff-a50b-8dc10155b33a-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:51.599306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:51.599278 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" Nov 03 16:26:51.599306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:51.599285 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw" event={"ID":"22ad70a9-6ab8-4eff-a50b-8dc10155b33a","Type":"ContainerDied","Data":"dde57287d5301ea4e24a8bd534ebe5ac7bdc5f1a97ecbff6940753ae97dae9d5"} Nov 03 16:26:51.599541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:51.599367 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde57287d5301ea4e24a8bd534ebe5ac7bdc5f1a97ecbff6940753ae97dae9d5" Nov 03 16:26:51.600676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:51.600658 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" Nov 03 16:26:51.600769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:51.600688 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx" event={"ID":"db7b15ab-7a30-45df-bb25-6f8ed973b0a0","Type":"ContainerDied","Data":"df1c46cd768f2c0dccccf55095656e0b162c04c50dea0890be35321a4234b293"} Nov 03 16:26:51.600769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:51.600736 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1c46cd768f2c0dccccf55095656e0b162c04c50dea0890be35321a4234b293" Nov 03 16:26:54.477283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:54.477235 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:55.483705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.483667 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg"] Nov 03 16:26:55.484324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484302 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ef3fcbb1-7a72-4bba-a064-423790337ce0" containerName="revoke-keys" Nov 03 16:26:55.484434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484325 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3fcbb1-7a72-4bba-a064-423790337ce0" containerName="revoke-keys" Nov 03 16:26:55.484434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484357 5946 cpu_manager.go:414] "containerMap: removing container" podUID="db7b15ab-7a30-45df-bb25-6f8ed973b0a0" containerName="revoke-keys" Nov 03 16:26:55.484434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484365 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7b15ab-7a30-45df-bb25-6f8ed973b0a0" containerName="revoke-keys" Nov 03 16:26:55.484434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484380 5946 cpu_manager.go:414] "containerMap: removing container" podUID="22ad70a9-6ab8-4eff-a50b-8dc10155b33a" containerName="revoke-keys" Nov 03 16:26:55.484434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484388 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ad70a9-6ab8-4eff-a50b-8dc10155b33a" containerName="revoke-keys" Nov 03 16:26:55.484679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484529 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3fcbb1-7a72-4bba-a064-423790337ce0" containerName="revoke-keys" Nov 03 16:26:55.484679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484543 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7b15ab-7a30-45df-bb25-6f8ed973b0a0" containerName="revoke-keys" Nov 03 16:26:55.484679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.484553 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ad70a9-6ab8-4eff-a50b-8dc10155b33a" containerName="revoke-keys" Nov 03 16:26:55.489080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.489058 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.540910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.540875 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.541061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.540935 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.541061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.541038 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrgl\" (UniqueName: \"kubernetes.io/projected/1b1678c3-9156-4c06-a700-89520590e7f6-kube-api-access-xlrgl\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.547627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.547601 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg"] Nov 03 16:26:55.595926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.595897 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:26:55.596115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.595933 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:26:55.596115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.595974 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:26:55.596115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.596025 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:26:55.642165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.642123 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.642363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.642201 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.642363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.642263 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrgl\" (UniqueName: \"kubernetes.io/projected/1b1678c3-9156-4c06-a700-89520590e7f6-kube-api-access-xlrgl\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.642935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.642914 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.644683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.644665 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.703702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.703670 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrgl\" (UniqueName: \"kubernetes.io/projected/1b1678c3-9156-4c06-a700-89520590e7f6-kube-api-access-xlrgl\") pod \"maas-api-revoke-sub-b23261a6-4v6kg\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:55.803656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:55.803577 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:56.181565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:56.181540 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg"] Nov 03 16:26:56.183449 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:56.183422 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1678c3_9156_4c06_a700_89520590e7f6.slice/crio-1ee470317c66d60b0c6ae6b3627c3861fde3b335426abf22a1ac1383da83dcfe WatchSource:0}: Error finding container 1ee470317c66d60b0c6ae6b3627c3861fde3b335426abf22a1ac1383da83dcfe: Status 404 returned error can't find the container with id 1ee470317c66d60b0c6ae6b3627c3861fde3b335426abf22a1ac1383da83dcfe Nov 03 16:26:56.631150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:56.631113 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" event={"ID":"1b1678c3-9156-4c06-a700-89520590e7f6","Type":"ContainerStarted","Data":"3c77dfbf27aeb5ee25f52d1fbb29697b959f5d16d76c6387a723cd71af83fa0e"} Nov 03 16:26:56.631625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:56.631176 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" event={"ID":"1b1678c3-9156-4c06-a700-89520590e7f6","Type":"ContainerStarted","Data":"1ee470317c66d60b0c6ae6b3627c3861fde3b335426abf22a1ac1383da83dcfe"} Nov 03 16:26:56.751835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:56.751780 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" podStartSLOduration=1.7517671959999999 podStartE2EDuration="1.751767196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:26:56.750741202 +0000 UTC m=+886.871794049" watchObservedRunningTime="2026-11-03 16:26:56.751767196 +0000 UTC m=+886.872820020" Nov 03 16:26:57.638641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:57.638604 5946 generic.go:356] "Generic (PLEG): container finished" podID="1b1678c3-9156-4c06-a700-89520590e7f6" containerID="3c77dfbf27aeb5ee25f52d1fbb29697b959f5d16d76c6387a723cd71af83fa0e" exitCode=0 Nov 03 16:26:57.639034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:57.638646 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" event={"ID":"1b1678c3-9156-4c06-a700-89520590e7f6","Type":"ContainerDied","Data":"3c77dfbf27aeb5ee25f52d1fbb29697b959f5d16d76c6387a723cd71af83fa0e"} Nov 03 16:26:58.779721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.779698 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:58.878182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.878147 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-xks-ca\") pod \"1b1678c3-9156-4c06-a700-89520590e7f6\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " Nov 03 16:26:58.878362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.878272 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1678c3-9156-4c06-a700-89520590e7f6-kube-api-access-xlrgl\" (UniqueName: \"kubernetes.io/projected/1b1678c3-9156-4c06-a700-89520590e7f6-kube-api-access-xlrgl\") pod \"1b1678c3-9156-4c06-a700-89520590e7f6\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " Nov 03 16:26:58.878362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.878296 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-openshift-ca\") pod \"1b1678c3-9156-4c06-a700-89520590e7f6\" (UID: \"1b1678c3-9156-4c06-a700-89520590e7f6\") " Nov 03 16:26:58.878699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.878675 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-openshift-ca" pod "1b1678c3-9156-4c06-a700-89520590e7f6" (UID: "1b1678c3-9156-4c06-a700-89520590e7f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:26:58.880361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.880339 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-xks-ca" pod "1b1678c3-9156-4c06-a700-89520590e7f6" (UID: "1b1678c3-9156-4c06-a700-89520590e7f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:26:58.880512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.880495 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1678c3-9156-4c06-a700-89520590e7f6-kube-api-access-xlrgl" pod "1b1678c3-9156-4c06-a700-89520590e7f6" (UID: "1b1678c3-9156-4c06-a700-89520590e7f6"). InnerVolumeSpecName "kube-api-access-xlrgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:26:58.980183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.980109 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:58.980183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.980138 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrgl\" (UniqueName: \"kubernetes.io/projected/1b1678c3-9156-4c06-a700-89520590e7f6-kube-api-access-xlrgl\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:58.980183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:58.980149 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1678c3-9156-4c06-a700-89520590e7f6-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:26:59.057055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.057017 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f"] Nov 03 16:26:59.057530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.057508 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1b1678c3-9156-4c06-a700-89520590e7f6" containerName="revoke-keys" Nov 03 16:26:59.057530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.057525 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1678c3-9156-4c06-a700-89520590e7f6" containerName="revoke-keys" Nov 03 16:26:59.057630 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.057611 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1678c3-9156-4c06-a700-89520590e7f6" containerName="revoke-keys" Nov 03 16:26:59.060807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.060792 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.129927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.129898 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f"] Nov 03 16:26:59.182790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.182754 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.182952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.182899 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.183048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.183025 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsw95\" (UniqueName: \"kubernetes.io/projected/23e88e5a-1f35-49f8-b354-b3f410fef5f0-kube-api-access-zsw95\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.283900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.283858 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.284088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.283941 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsw95\" (UniqueName: \"kubernetes.io/projected/23e88e5a-1f35-49f8-b354-b3f410fef5f0-kube-api-access-zsw95\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.284088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.283988 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.284640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.284620 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.286438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.286414 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.345955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.345925 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsw95\" (UniqueName: \"kubernetes.io/projected/23e88e5a-1f35-49f8-b354-b3f410fef5f0-kube-api-access-zsw95\") pod \"maas-api-revoke-sub-1b004db6-mz97f\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.362295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.362264 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:59.376191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.376162 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:26:59.653450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.653348 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" event={"ID":"1b1678c3-9156-4c06-a700-89520590e7f6","Type":"ContainerDied","Data":"1ee470317c66d60b0c6ae6b3627c3861fde3b335426abf22a1ac1383da83dcfe"} Nov 03 16:26:59.653450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.653416 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ee470317c66d60b0c6ae6b3627c3861fde3b335426abf22a1ac1383da83dcfe" Nov 03 16:26:59.653691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.653478 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg" Nov 03 16:26:59.763326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:26:59.763301 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f"] Nov 03 16:26:59.764784 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:26:59.764755 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e88e5a_1f35_49f8_b354_b3f410fef5f0.slice/crio-f6f0e0500a3219ca8ab6631d5cc43c4accab87a9c3669c23ae52cf319705c04b WatchSource:0}: Error finding container f6f0e0500a3219ca8ab6631d5cc43c4accab87a9c3669c23ae52cf319705c04b: Status 404 returned error can't find the container with id f6f0e0500a3219ca8ab6631d5cc43c4accab87a9c3669c23ae52cf319705c04b Nov 03 16:27:00.116728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:00.116692 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:27:00.660032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:00.659998 5946 generic.go:356] "Generic (PLEG): container finished" podID="23e88e5a-1f35-49f8-b354-b3f410fef5f0" containerID="34e1339d1c8bbf3f0c668f36facbe5e49e4c0c34589a095f471fd60346b9f8de" exitCode=0 Nov 03 16:27:00.660166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:00.660084 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" event={"ID":"23e88e5a-1f35-49f8-b354-b3f410fef5f0","Type":"ContainerDied","Data":"34e1339d1c8bbf3f0c668f36facbe5e49e4c0c34589a095f471fd60346b9f8de"} Nov 03 16:27:00.660166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:00.660144 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" event={"ID":"23e88e5a-1f35-49f8-b354-b3f410fef5f0","Type":"ContainerStarted","Data":"f6f0e0500a3219ca8ab6631d5cc43c4accab87a9c3669c23ae52cf319705c04b"} Nov 03 16:27:01.897518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:01.897488 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:27:02.013993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.013961 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-xks-ca\") pod \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " Nov 03 16:27:02.014153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.014084 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e88e5a-1f35-49f8-b354-b3f410fef5f0-kube-api-access-zsw95\" (UniqueName: \"kubernetes.io/projected/23e88e5a-1f35-49f8-b354-b3f410fef5f0-kube-api-access-zsw95\") pod \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " Nov 03 16:27:02.014237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.014179 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-openshift-ca\") pod \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\" (UID: \"23e88e5a-1f35-49f8-b354-b3f410fef5f0\") " Nov 03 16:27:02.015049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.014579 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-openshift-ca" pod "23e88e5a-1f35-49f8-b354-b3f410fef5f0" (UID: "23e88e5a-1f35-49f8-b354-b3f410fef5f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:02.015049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.014717 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:02.016574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.016544 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e88e5a-1f35-49f8-b354-b3f410fef5f0-kube-api-access-zsw95" pod "23e88e5a-1f35-49f8-b354-b3f410fef5f0" (UID: "23e88e5a-1f35-49f8-b354-b3f410fef5f0"). InnerVolumeSpecName "kube-api-access-zsw95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:02.016730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.016711 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-xks-ca" pod "23e88e5a-1f35-49f8-b354-b3f410fef5f0" (UID: "23e88e5a-1f35-49f8-b354-b3f410fef5f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:02.115926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.115889 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsw95\" (UniqueName: \"kubernetes.io/projected/23e88e5a-1f35-49f8-b354-b3f410fef5f0-kube-api-access-zsw95\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:02.115926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.115928 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e88e5a-1f35-49f8-b354-b3f410fef5f0-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:02.319037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.318948 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj"] Nov 03 16:27:02.319690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.319666 5946 cpu_manager.go:414] "containerMap: removing container" podUID="23e88e5a-1f35-49f8-b354-b3f410fef5f0" containerName="revoke-keys" Nov 03 16:27:02.319783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.319693 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e88e5a-1f35-49f8-b354-b3f410fef5f0" containerName="revoke-keys" Nov 03 16:27:02.319878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.319862 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e88e5a-1f35-49f8-b354-b3f410fef5f0" containerName="revoke-keys" Nov 03 16:27:02.323725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.323698 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.383223 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.383176 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj"] Nov 03 16:27:02.419117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.419073 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.419371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.419343 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.419459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.419392 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsbp\" (UniqueName: \"kubernetes.io/projected/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-kube-api-access-mwsbp\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.520549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.520519 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.520549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.520554 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsbp\" (UniqueName: \"kubernetes.io/projected/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-kube-api-access-mwsbp\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.520820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.520609 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.521400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.521367 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.523279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.523252 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.582803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.582727 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsbp\" (UniqueName: \"kubernetes.io/projected/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-kube-api-access-mwsbp\") pod \"maas-api-revoke-sub-69a7fd45-t7gfj\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.644930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.644853 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:02.680794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.680763 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" Nov 03 16:27:02.680939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.680836 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f" event={"ID":"23e88e5a-1f35-49f8-b354-b3f410fef5f0","Type":"ContainerDied","Data":"f6f0e0500a3219ca8ab6631d5cc43c4accab87a9c3669c23ae52cf319705c04b"} Nov 03 16:27:02.680939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.680926 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6f0e0500a3219ca8ab6631d5cc43c4accab87a9c3669c23ae52cf319705c04b" Nov 03 16:27:02.785693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.785659 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn"] Nov 03 16:27:02.791355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.791326 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.827023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.826977 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.827321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.827290 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.827433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.827362 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mfx\" (UniqueName: \"kubernetes.io/projected/575a9e39-604f-4f27-a77f-2132e73870d8-kube-api-access-d9mfx\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.862437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.862355 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn"] Nov 03 16:27:02.928182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.928141 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.928584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.928297 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.928584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.928340 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mfx\" (UniqueName: \"kubernetes.io/projected/575a9e39-604f-4f27-a77f-2132e73870d8-kube-api-access-d9mfx\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.930718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.929534 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.932183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.932158 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:02.989675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:02.989643 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mfx\" (UniqueName: \"kubernetes.io/projected/575a9e39-604f-4f27-a77f-2132e73870d8-kube-api-access-d9mfx\") pod \"maas-api-revoke-sub-30c6ce0f-6vfjn\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:03.052498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:03.052465 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj"] Nov 03 16:27:03.059108 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:03.059072 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7ca9ca_ceaf_46d9_b6fb_c4e6f9dcff05.slice/crio-9096892cfd0be328a1c78270e22329d54e2ede05693863c4a5fa9cab0d0b410b WatchSource:0}: Error finding container 9096892cfd0be328a1c78270e22329d54e2ede05693863c4a5fa9cab0d0b410b: Status 404 returned error can't find the container with id 9096892cfd0be328a1c78270e22329d54e2ede05693863c4a5fa9cab0d0b410b Nov 03 16:27:03.128861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:03.128485 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:03.529691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:03.529669 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn"] Nov 03 16:27:03.531073 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:03.531047 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575a9e39_604f_4f27_a77f_2132e73870d8.slice/crio-da7f63a38059bf49ceb9c4c7a26467c33b4445dd2f6042a9e24f9603493b69e7 WatchSource:0}: Error finding container da7f63a38059bf49ceb9c4c7a26467c33b4445dd2f6042a9e24f9603493b69e7: Status 404 returned error can't find the container with id da7f63a38059bf49ceb9c4c7a26467c33b4445dd2f6042a9e24f9603493b69e7 Nov 03 16:27:03.688951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:03.688914 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" event={"ID":"575a9e39-604f-4f27-a77f-2132e73870d8","Type":"ContainerStarted","Data":"da7f63a38059bf49ceb9c4c7a26467c33b4445dd2f6042a9e24f9603493b69e7"} Nov 03 16:27:03.690506 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:03.690472 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" event={"ID":"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05","Type":"ContainerStarted","Data":"720f869459e853ea13553a68ed62112337dd004d8d321ec2786ae7d89312b1cc"} Nov 03 16:27:03.690642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:03.690527 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" event={"ID":"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05","Type":"ContainerStarted","Data":"9096892cfd0be328a1c78270e22329d54e2ede05693863c4a5fa9cab0d0b410b"} Nov 03 16:27:03.810796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:03.810699 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" podStartSLOduration=1.810686276 podStartE2EDuration="1.810686276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:03.809830107 +0000 UTC m=+893.930882930" watchObservedRunningTime="2026-11-03 16:27:03.810686276 +0000 UTC m=+893.931739098" Nov 03 16:27:04.260836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:04.260656 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:04.698534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:04.698449 5946 generic.go:356] "Generic (PLEG): container finished" podID="575a9e39-604f-4f27-a77f-2132e73870d8" containerID="aaf8c316ffcc392f53fc4f10d632ba8d27ae8419b23a8325e4550ebf9c257f47" exitCode=0 Nov 03 16:27:04.698683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:04.698532 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" event={"ID":"575a9e39-604f-4f27-a77f-2132e73870d8","Type":"ContainerDied","Data":"aaf8c316ffcc392f53fc4f10d632ba8d27ae8419b23a8325e4550ebf9c257f47"} Nov 03 16:27:04.700148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:04.700119 5946 generic.go:356] "Generic (PLEG): container finished" podID="5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" containerID="720f869459e853ea13553a68ed62112337dd004d8d321ec2786ae7d89312b1cc" exitCode=0 Nov 03 16:27:04.700301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:04.700176 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" event={"ID":"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05","Type":"ContainerDied","Data":"720f869459e853ea13553a68ed62112337dd004d8d321ec2786ae7d89312b1cc"} Nov 03 16:27:05.866184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.866158 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:05.896392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.896367 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:05.963114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963023 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-xks-ca\") pod \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " Nov 03 16:27:05.963114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963083 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-kube-api-access-mwsbp\" (UniqueName: \"kubernetes.io/projected/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-kube-api-access-mwsbp\") pod \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " Nov 03 16:27:05.963374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963258 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-openshift-ca\") pod \"575a9e39-604f-4f27-a77f-2132e73870d8\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " Nov 03 16:27:05.963374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963303 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-openshift-ca\") pod \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\" (UID: \"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05\") " Nov 03 16:27:05.963374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963340 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575a9e39-604f-4f27-a77f-2132e73870d8-kube-api-access-d9mfx\" (UniqueName: \"kubernetes.io/projected/575a9e39-604f-4f27-a77f-2132e73870d8-kube-api-access-d9mfx\") pod \"575a9e39-604f-4f27-a77f-2132e73870d8\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " Nov 03 16:27:05.963521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963403 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-xks-ca\") pod \"575a9e39-604f-4f27-a77f-2132e73870d8\" (UID: \"575a9e39-604f-4f27-a77f-2132e73870d8\") " Nov 03 16:27:05.963667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963639 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-openshift-ca" pod "575a9e39-604f-4f27-a77f-2132e73870d8" (UID: "575a9e39-604f-4f27-a77f-2132e73870d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:05.963764 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963661 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-openshift-ca" pod "5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" (UID: "5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:05.963876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963854 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:05.963876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.963878 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:05.965355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.965328 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-xks-ca" pod "5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" (UID: "5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:05.965531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.965509 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575a9e39-604f-4f27-a77f-2132e73870d8-kube-api-access-d9mfx" pod "575a9e39-604f-4f27-a77f-2132e73870d8" (UID: "575a9e39-604f-4f27-a77f-2132e73870d8"). InnerVolumeSpecName "kube-api-access-d9mfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:05.965666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.965651 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-kube-api-access-mwsbp" pod "5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" (UID: "5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05"). InnerVolumeSpecName "kube-api-access-mwsbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:05.965966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:05.965950 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-xks-ca" pod "575a9e39-604f-4f27-a77f-2132e73870d8" (UID: "575a9e39-604f-4f27-a77f-2132e73870d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:06.065341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.065309 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwsbp\" (UniqueName: \"kubernetes.io/projected/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-kube-api-access-mwsbp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:06.065341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.065343 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mfx\" (UniqueName: \"kubernetes.io/projected/575a9e39-604f-4f27-a77f-2132e73870d8-kube-api-access-d9mfx\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:06.065535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.065359 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a9e39-604f-4f27-a77f-2132e73870d8-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:06.065535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.065375 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:06.713758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.713728 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" Nov 03 16:27:06.714064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.713732 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn" event={"ID":"575a9e39-604f-4f27-a77f-2132e73870d8","Type":"ContainerDied","Data":"da7f63a38059bf49ceb9c4c7a26467c33b4445dd2f6042a9e24f9603493b69e7"} Nov 03 16:27:06.714064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.713857 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da7f63a38059bf49ceb9c4c7a26467c33b4445dd2f6042a9e24f9603493b69e7" Nov 03 16:27:06.715158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.715137 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" Nov 03 16:27:06.715297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.715177 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj" event={"ID":"5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05","Type":"ContainerDied","Data":"9096892cfd0be328a1c78270e22329d54e2ede05693863c4a5fa9cab0d0b410b"} Nov 03 16:27:06.715297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:06.715238 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9096892cfd0be328a1c78270e22329d54e2ede05693863c4a5fa9cab0d0b410b" Nov 03 16:27:09.004489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:09.004450 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:10.113015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:10.112986 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:27:10.114740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:10.114721 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:27:12.584969 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:27:12.584934 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:27:13.882862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:13.882831 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:16.253932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.253897 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx"] Nov 03 16:27:16.254411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.254361 5946 cpu_manager.go:414] "containerMap: removing container" podUID="5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" containerName="revoke-keys" Nov 03 16:27:16.254411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.254374 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" containerName="revoke-keys" Nov 03 16:27:16.254411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.254384 5946 cpu_manager.go:414] "containerMap: removing container" podUID="575a9e39-604f-4f27-a77f-2132e73870d8" containerName="revoke-keys" Nov 03 16:27:16.254411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.254389 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a9e39-604f-4f27-a77f-2132e73870d8" containerName="revoke-keys" Nov 03 16:27:16.254569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.254535 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" containerName="revoke-keys" Nov 03 16:27:16.254569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.254548 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a9e39-604f-4f27-a77f-2132e73870d8" containerName="revoke-keys" Nov 03 16:27:16.259356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.259340 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.319285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.319255 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx"] Nov 03 16:27:16.366440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.366404 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:27:16.366440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.366434 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:27:16.366688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.366497 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:27:16.366688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.366642 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:27:16.371417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.371397 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.371500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.371448 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.371580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.371564 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvhw\" (UniqueName: \"kubernetes.io/projected/184c6de3-969d-4432-bd5d-62dc8144dd81-kube-api-access-drvhw\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.472309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.472274 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.472502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.472335 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.472502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.472450 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvhw\" (UniqueName: \"kubernetes.io/projected/184c6de3-969d-4432-bd5d-62dc8144dd81-kube-api-access-drvhw\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.472928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.472908 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.474895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.474875 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.533711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.533639 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvhw\" (UniqueName: \"kubernetes.io/projected/184c6de3-969d-4432-bd5d-62dc8144dd81-kube-api-access-drvhw\") pod \"maas-api-revoke-sub-478e8921-s48sx\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.573542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.573503 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:16.955650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:16.955625 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx"] Nov 03 16:27:16.957627 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:16.957593 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184c6de3_969d_4432_bd5d_62dc8144dd81.slice/crio-763442e0105a2944020b610efd694102c9d27f4b045a2defa8badbd42518c160 WatchSource:0}: Error finding container 763442e0105a2944020b610efd694102c9d27f4b045a2defa8badbd42518c160: Status 404 returned error can't find the container with id 763442e0105a2944020b610efd694102c9d27f4b045a2defa8badbd42518c160 Nov 03 16:27:17.448057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:17.448026 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:17.789036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:17.789003 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" event={"ID":"184c6de3-969d-4432-bd5d-62dc8144dd81","Type":"ContainerStarted","Data":"b1d54c8692b07ce93b9b669345b8a8b36906e7b41ad644d073188409e23c02a1"} Nov 03 16:27:17.789152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:17.789065 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" event={"ID":"184c6de3-969d-4432-bd5d-62dc8144dd81","Type":"ContainerStarted","Data":"763442e0105a2944020b610efd694102c9d27f4b045a2defa8badbd42518c160"} Nov 03 16:27:18.676599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.676565 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6"] Nov 03 16:27:18.680705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.680680 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.697503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.697475 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:18.777701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.777668 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6"] Nov 03 16:27:18.797944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.797905 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s464\" (UniqueName: \"kubernetes.io/projected/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-kube-api-access-4s464\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.798129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.798042 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.798276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.798129 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.802191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.802164 5946 generic.go:356] "Generic (PLEG): container finished" podID="184c6de3-969d-4432-bd5d-62dc8144dd81" containerID="b1d54c8692b07ce93b9b669345b8a8b36906e7b41ad644d073188409e23c02a1" exitCode=0 Nov 03 16:27:18.802333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.802242 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" event={"ID":"184c6de3-969d-4432-bd5d-62dc8144dd81","Type":"ContainerDied","Data":"b1d54c8692b07ce93b9b669345b8a8b36906e7b41ad644d073188409e23c02a1"} Nov 03 16:27:18.899902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.899872 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.900049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.899933 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s464\" (UniqueName: \"kubernetes.io/projected/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-kube-api-access-4s464\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.900049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.899989 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.900612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.900588 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.902461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.902435 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.943945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.943895 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:18.960016 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.959989 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s464\" (UniqueName: \"kubernetes.io/projected/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-kube-api-access-4s464\") pod \"maas-api-revoke-sub-c37c1961-mgsh6\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:18.996726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:18.996692 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:19.101359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.101329 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184c6de3-969d-4432-bd5d-62dc8144dd81-kube-api-access-drvhw\" (UniqueName: \"kubernetes.io/projected/184c6de3-969d-4432-bd5d-62dc8144dd81-kube-api-access-drvhw\") pod \"184c6de3-969d-4432-bd5d-62dc8144dd81\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " Nov 03 16:27:19.101513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.101429 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-openshift-ca\") pod \"184c6de3-969d-4432-bd5d-62dc8144dd81\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " Nov 03 16:27:19.101513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.101501 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-xks-ca\") pod \"184c6de3-969d-4432-bd5d-62dc8144dd81\" (UID: \"184c6de3-969d-4432-bd5d-62dc8144dd81\") " Nov 03 16:27:19.101855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.101827 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-openshift-ca" pod "184c6de3-969d-4432-bd5d-62dc8144dd81" (UID: "184c6de3-969d-4432-bd5d-62dc8144dd81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:19.101989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.101957 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:19.103645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.103620 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184c6de3-969d-4432-bd5d-62dc8144dd81-kube-api-access-drvhw" pod "184c6de3-969d-4432-bd5d-62dc8144dd81" (UID: "184c6de3-969d-4432-bd5d-62dc8144dd81"). InnerVolumeSpecName "kube-api-access-drvhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:19.103984 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.103959 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-xks-ca" pod "184c6de3-969d-4432-bd5d-62dc8144dd81" (UID: "184c6de3-969d-4432-bd5d-62dc8144dd81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:19.202635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.202549 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c6de3-969d-4432-bd5d-62dc8144dd81-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:19.202635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.202580 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drvhw\" (UniqueName: \"kubernetes.io/projected/184c6de3-969d-4432-bd5d-62dc8144dd81-kube-api-access-drvhw\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:19.277975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.277902 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4"] Nov 03 16:27:19.278513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.278489 5946 cpu_manager.go:414] "containerMap: removing container" podUID="184c6de3-969d-4432-bd5d-62dc8144dd81" containerName="revoke-keys" Nov 03 16:27:19.278513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.278513 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184c6de3-969d-4432-bd5d-62dc8144dd81" containerName="revoke-keys" Nov 03 16:27:19.278698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.278684 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="184c6de3-969d-4432-bd5d-62dc8144dd81" containerName="revoke-keys" Nov 03 16:27:19.281993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.281971 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.371368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.371332 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4"] Nov 03 16:27:19.381194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.381166 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6"] Nov 03 16:27:19.381547 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:19.381514 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1c4ff8_70b5_40c9_80c2_009b8afc7933.slice/crio-c5bff0da163172d1b3497a4b668b4211ffdf8aa2bdadfc4895a022d030f2c940 WatchSource:0}: Error finding container c5bff0da163172d1b3497a4b668b4211ffdf8aa2bdadfc4895a022d030f2c940: Status 404 returned error can't find the container with id c5bff0da163172d1b3497a4b668b4211ffdf8aa2bdadfc4895a022d030f2c940 Nov 03 16:27:19.404890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.404863 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.404902 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.405012 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.405054 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.405138 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.405249 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.405293 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.405395 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.405470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.405423 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4d8\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-kube-api-access-5w4d8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506294 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506340 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4d8\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-kube-api-access-5w4d8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506376 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506409 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506444 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506576 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506787 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506810 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506853 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.506979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.506951 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.507044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.507029 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.507096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.507088 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.507413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.507392 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.507605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.507584 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.508762 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.508743 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.509337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.509320 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.568162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.568136 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.568347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.568326 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4d8\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-kube-api-access-5w4d8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qpmp4\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.595818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.595794 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:19.810816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.810736 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" event={"ID":"184c6de3-969d-4432-bd5d-62dc8144dd81","Type":"ContainerDied","Data":"763442e0105a2944020b610efd694102c9d27f4b045a2defa8badbd42518c160"} Nov 03 16:27:19.810816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.810790 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763442e0105a2944020b610efd694102c9d27f4b045a2defa8badbd42518c160" Nov 03 16:27:19.810816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.810743 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx" Nov 03 16:27:19.812931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.812908 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" event={"ID":"1b1c4ff8-70b5-40c9-80c2-009b8afc7933","Type":"ContainerStarted","Data":"898c93f24b29721e2feb1c7fe452d3881317cad690921dcbc12f2087f3c1ba1c"} Nov 03 16:27:19.813062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.812947 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" event={"ID":"1b1c4ff8-70b5-40c9-80c2-009b8afc7933","Type":"ContainerStarted","Data":"c5bff0da163172d1b3497a4b668b4211ffdf8aa2bdadfc4895a022d030f2c940"} Nov 03 16:27:19.933368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.933316 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" podStartSLOduration=1.933302308 podStartE2EDuration="1.933302308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:19.931713078 +0000 UTC m=+910.052765914" watchObservedRunningTime="2026-11-03 16:27:19.933302308 +0000 UTC m=+910.054355130" Nov 03 16:27:19.963380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.963346 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz"] Nov 03 16:27:19.967088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:19.967072 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.026248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.026194 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz"] Nov 03 16:27:20.032457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.032428 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4"] Nov 03 16:27:20.033084 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:20.033053 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56316bb2_7f63_4700_a987_96d2ca2cb2fe.slice/crio-a83af99c88e7f38adc9a8f461a28797707dc66d9ce6caf97da0a87308e6be9a3 WatchSource:0}: Error finding container a83af99c88e7f38adc9a8f461a28797707dc66d9ce6caf97da0a87308e6be9a3: Status 404 returned error can't find the container with id a83af99c88e7f38adc9a8f461a28797707dc66d9ce6caf97da0a87308e6be9a3 Nov 03 16:27:20.035945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.035853 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:20.035945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.035932 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:20.036105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.035971 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:20.114027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.113994 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.114199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.114073 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjf7\" (UniqueName: \"kubernetes.io/projected/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-kube-api-access-mxjf7\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.114292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.114267 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.215129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.215086 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.215335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.215155 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjf7\" (UniqueName: \"kubernetes.io/projected/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-kube-api-access-mxjf7\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.215335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.215250 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.215810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.215785 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.218059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.218035 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.277036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.276999 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjf7\" (UniqueName: \"kubernetes.io/projected/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-kube-api-access-mxjf7\") pod \"maas-api-revoke-sub-efc8fd95-jt6kz\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.282646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.282619 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:20.526766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.526724 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q"] Nov 03 16:27:20.530772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.530755 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.590901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.590821 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q"] Nov 03 16:27:20.677042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.677015 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz"] Nov 03 16:27:20.678138 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:20.678113 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e94dafd_fee1_4bf6_b654_1c2ff9054d91.slice/crio-d4c3efc4c20770a4461b0c5eaf56c77a7fb0016e7e6297cb44adad8cae3bafcc WatchSource:0}: Error finding container d4c3efc4c20770a4461b0c5eaf56c77a7fb0016e7e6297cb44adad8cae3bafcc: Status 404 returned error can't find the container with id d4c3efc4c20770a4461b0c5eaf56c77a7fb0016e7e6297cb44adad8cae3bafcc Nov 03 16:27:20.719789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.719755 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.719971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.719924 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svgw2\" (UniqueName: \"kubernetes.io/projected/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-kube-api-access-svgw2\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.720038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.719982 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.823263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.822041 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svgw2\" (UniqueName: \"kubernetes.io/projected/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-kube-api-access-svgw2\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.823263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.822505 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.823263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.822553 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.823822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.823603 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.826505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.826464 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.840830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.840770 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" event={"ID":"3e94dafd-fee1-4bf6-b654-1c2ff9054d91","Type":"ContainerStarted","Data":"d4c3efc4c20770a4461b0c5eaf56c77a7fb0016e7e6297cb44adad8cae3bafcc"} Nov 03 16:27:20.844283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.842827 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" event={"ID":"56316bb2-7f63-4700-a987-96d2ca2cb2fe","Type":"ContainerStarted","Data":"966698430d884b2264b06d9a4c3a494c3fc3f0d79ae4785b4ee089b62a5ad25b"} Nov 03 16:27:20.844283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.842878 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" event={"ID":"56316bb2-7f63-4700-a987-96d2ca2cb2fe","Type":"ContainerStarted","Data":"a83af99c88e7f38adc9a8f461a28797707dc66d9ce6caf97da0a87308e6be9a3"} Nov 03 16:27:20.845884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.845854 5946 generic.go:356] "Generic (PLEG): container finished" podID="1b1c4ff8-70b5-40c9-80c2-009b8afc7933" containerID="898c93f24b29721e2feb1c7fe452d3881317cad690921dcbc12f2087f3c1ba1c" exitCode=0 Nov 03 16:27:20.846027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.845981 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" event={"ID":"1b1c4ff8-70b5-40c9-80c2-009b8afc7933","Type":"ContainerDied","Data":"898c93f24b29721e2feb1c7fe452d3881317cad690921dcbc12f2087f3c1ba1c"} Nov 03 16:27:20.883342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.883316 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svgw2\" (UniqueName: \"kubernetes.io/projected/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-kube-api-access-svgw2\") pod \"maas-api-revoke-sub-7eeb0724-jbc4q\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:20.968538 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:20.968486 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podStartSLOduration=1.9684712580000001 podStartE2EDuration="1.968471258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:20.965662368 +0000 UTC m=+911.086715201" watchObservedRunningTime="2026-11-03 16:27:20.968471258 +0000 UTC m=+911.089524083" Nov 03 16:27:21.144483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.144397 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:21.545268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.545231 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q"] Nov 03 16:27:21.545980 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:21.545945 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9f0a0b_91ae_4d58_b97e_53a436972c3e.slice/crio-c654a7c16303870f79fa0a18f66fa6a8c0fe4e2a715f86f8c71e7c85bf18dc40 WatchSource:0}: Error finding container c654a7c16303870f79fa0a18f66fa6a8c0fe4e2a715f86f8c71e7c85bf18dc40: Status 404 returned error can't find the container with id c654a7c16303870f79fa0a18f66fa6a8c0fe4e2a715f86f8c71e7c85bf18dc40 Nov 03 16:27:21.596283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.596254 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:21.597787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.597759 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qpmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:27:21.597901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.597814 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 16:27:21.853451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.853415 5946 generic.go:356] "Generic (PLEG): container finished" podID="3e94dafd-fee1-4bf6-b654-1c2ff9054d91" containerID="8bd0561007a3197340cbff60e61f06bb49a97dac9b70120f1b1aaa583a67512d" exitCode=0 Nov 03 16:27:21.853879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.853500 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" event={"ID":"3e94dafd-fee1-4bf6-b654-1c2ff9054d91","Type":"ContainerDied","Data":"8bd0561007a3197340cbff60e61f06bb49a97dac9b70120f1b1aaa583a67512d"} Nov 03 16:27:21.857940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:21.857655 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" event={"ID":"3c9f0a0b-91ae-4d58-b97e-53a436972c3e","Type":"ContainerStarted","Data":"c654a7c16303870f79fa0a18f66fa6a8c0fe4e2a715f86f8c71e7c85bf18dc40"} Nov 03 16:27:22.076829 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.076756 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:22.158625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.158583 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-xks-ca\") pod \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " Nov 03 16:27:22.158784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.158662 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-openshift-ca\") pod \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " Nov 03 16:27:22.158784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.158776 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-kube-api-access-4s464\" (UniqueName: \"kubernetes.io/projected/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-kube-api-access-4s464\") pod \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\" (UID: \"1b1c4ff8-70b5-40c9-80c2-009b8afc7933\") " Nov 03 16:27:22.159036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.159005 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-openshift-ca" pod "1b1c4ff8-70b5-40c9-80c2-009b8afc7933" (UID: "1b1c4ff8-70b5-40c9-80c2-009b8afc7933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:22.159222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.159184 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:22.160935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.160912 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-xks-ca" pod "1b1c4ff8-70b5-40c9-80c2-009b8afc7933" (UID: "1b1c4ff8-70b5-40c9-80c2-009b8afc7933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:22.161030 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.160971 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-kube-api-access-4s464" pod "1b1c4ff8-70b5-40c9-80c2-009b8afc7933" (UID: "1b1c4ff8-70b5-40c9-80c2-009b8afc7933"). InnerVolumeSpecName "kube-api-access-4s464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:22.259975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.259930 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s464\" (UniqueName: \"kubernetes.io/projected/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-kube-api-access-4s464\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:22.259975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.259972 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1c4ff8-70b5-40c9-80c2-009b8afc7933-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:22.596514 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.596485 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qpmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:27:22.596675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.596535 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 16:27:22.867453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.867360 5946 generic.go:356] "Generic (PLEG): container finished" podID="3c9f0a0b-91ae-4d58-b97e-53a436972c3e" containerID="020b7fe15421a2694abe3c6c9ea72ea2a0274e8f587cf2db1c6dd3a9069d9adf" exitCode=0 Nov 03 16:27:22.867840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.867458 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" event={"ID":"3c9f0a0b-91ae-4d58-b97e-53a436972c3e","Type":"ContainerDied","Data":"020b7fe15421a2694abe3c6c9ea72ea2a0274e8f587cf2db1c6dd3a9069d9adf"} Nov 03 16:27:22.870473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.870397 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" event={"ID":"1b1c4ff8-70b5-40c9-80c2-009b8afc7933","Type":"ContainerDied","Data":"c5bff0da163172d1b3497a4b668b4211ffdf8aa2bdadfc4895a022d030f2c940"} Nov 03 16:27:22.870473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.870448 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bff0da163172d1b3497a4b668b4211ffdf8aa2bdadfc4895a022d030f2c940" Nov 03 16:27:22.871357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:22.871187 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6" Nov 03 16:27:23.078188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.078166 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:23.172362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.172254 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-xks-ca\") pod \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " Nov 03 16:27:23.172544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.172379 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-kube-api-access-mxjf7\" (UniqueName: \"kubernetes.io/projected/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-kube-api-access-mxjf7\") pod \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " Nov 03 16:27:23.175049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.175014 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-kube-api-access-mxjf7" pod "3e94dafd-fee1-4bf6-b654-1c2ff9054d91" (UID: "3e94dafd-fee1-4bf6-b654-1c2ff9054d91"). InnerVolumeSpecName "kube-api-access-mxjf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:23.175345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.175314 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-xks-ca" pod "3e94dafd-fee1-4bf6-b654-1c2ff9054d91" (UID: "3e94dafd-fee1-4bf6-b654-1c2ff9054d91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:23.278248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.273291 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-openshift-ca\") pod \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\" (UID: \"3e94dafd-fee1-4bf6-b654-1c2ff9054d91\") " Nov 03 16:27:23.278248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.273705 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjf7\" (UniqueName: \"kubernetes.io/projected/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-kube-api-access-mxjf7\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:23.278248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.273727 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:23.278248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.274030 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-openshift-ca" pod "3e94dafd-fee1-4bf6-b654-1c2ff9054d91" (UID: "3e94dafd-fee1-4bf6-b654-1c2ff9054d91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:23.374974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.374940 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e94dafd-fee1-4bf6-b654-1c2ff9054d91-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:23.548572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.548534 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:23.596923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.596887 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qpmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:27:23.597102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.596951 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 16:27:23.877449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.877371 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" Nov 03 16:27:23.877449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.877407 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz" event={"ID":"3e94dafd-fee1-4bf6-b654-1c2ff9054d91","Type":"ContainerDied","Data":"d4c3efc4c20770a4461b0c5eaf56c77a7fb0016e7e6297cb44adad8cae3bafcc"} Nov 03 16:27:23.877873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:23.877494 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4c3efc4c20770a4461b0c5eaf56c77a7fb0016e7e6297cb44adad8cae3bafcc" Nov 03 16:27:24.066523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.066487 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:24.082905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.082876 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-openshift-ca\") pod \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " Nov 03 16:27:24.083077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.082956 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-kube-api-access-svgw2\" (UniqueName: \"kubernetes.io/projected/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-kube-api-access-svgw2\") pod \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " Nov 03 16:27:24.083077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.082994 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-xks-ca\") pod \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\" (UID: \"3c9f0a0b-91ae-4d58-b97e-53a436972c3e\") " Nov 03 16:27:24.083303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.083279 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-openshift-ca" pod "3c9f0a0b-91ae-4d58-b97e-53a436972c3e" (UID: "3c9f0a0b-91ae-4d58-b97e-53a436972c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:24.083421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.083404 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:24.085252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.085201 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-xks-ca" pod "3c9f0a0b-91ae-4d58-b97e-53a436972c3e" (UID: "3c9f0a0b-91ae-4d58-b97e-53a436972c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:24.085353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.085275 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-kube-api-access-svgw2" pod "3c9f0a0b-91ae-4d58-b97e-53a436972c3e" (UID: "3c9f0a0b-91ae-4d58-b97e-53a436972c3e"). InnerVolumeSpecName "kube-api-access-svgw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:24.184582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.184497 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:24.184582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.184526 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svgw2\" (UniqueName: \"kubernetes.io/projected/3c9f0a0b-91ae-4d58-b97e-53a436972c3e-kube-api-access-svgw2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:24.597013 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.596976 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qpmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:27:24.597169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.597033 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 16:27:24.884720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.884641 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" Nov 03 16:27:24.884720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.884669 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q" event={"ID":"3c9f0a0b-91ae-4d58-b97e-53a436972c3e","Type":"ContainerDied","Data":"c654a7c16303870f79fa0a18f66fa6a8c0fe4e2a715f86f8c71e7c85bf18dc40"} Nov 03 16:27:24.884720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:24.884721 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c654a7c16303870f79fa0a18f66fa6a8c0fe4e2a715f86f8c71e7c85bf18dc40" Nov 03 16:27:25.295886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.295841 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl"] Nov 03 16:27:25.296310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296297 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3e94dafd-fee1-4bf6-b654-1c2ff9054d91" containerName="revoke-keys" Nov 03 16:27:25.296310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296310 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e94dafd-fee1-4bf6-b654-1c2ff9054d91" containerName="revoke-keys" Nov 03 16:27:25.296418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296332 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1b1c4ff8-70b5-40c9-80c2-009b8afc7933" containerName="revoke-keys" Nov 03 16:27:25.296418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296337 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1c4ff8-70b5-40c9-80c2-009b8afc7933" containerName="revoke-keys" Nov 03 16:27:25.296418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296349 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3c9f0a0b-91ae-4d58-b97e-53a436972c3e" containerName="revoke-keys" Nov 03 16:27:25.296418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296354 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9f0a0b-91ae-4d58-b97e-53a436972c3e" containerName="revoke-keys" Nov 03 16:27:25.296552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296428 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e94dafd-fee1-4bf6-b654-1c2ff9054d91" containerName="revoke-keys" Nov 03 16:27:25.296552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296439 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1c4ff8-70b5-40c9-80c2-009b8afc7933" containerName="revoke-keys" Nov 03 16:27:25.296552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.296448 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9f0a0b-91ae-4d58-b97e-53a436972c3e" containerName="revoke-keys" Nov 03 16:27:25.299735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.299720 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.361686 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.361656 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl"] Nov 03 16:27:25.398454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.398421 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.398454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.398461 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwf4\" (UniqueName: \"kubernetes.io/projected/750a6935-778a-4b2f-b049-a9e07ec19b03-kube-api-access-gdwf4\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.398656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.398555 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.408132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.408107 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:27:25.408132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.408132 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:27:25.408330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.408147 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:27:25.408330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.408194 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:27:25.499999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.499960 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.499999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.500000 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwf4\" (UniqueName: \"kubernetes.io/projected/750a6935-778a-4b2f-b049-a9e07ec19b03-kube-api-access-gdwf4\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.500259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.500056 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.500801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.500783 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.502581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.502563 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.561839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.561756 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwf4\" (UniqueName: \"kubernetes.io/projected/750a6935-778a-4b2f-b049-a9e07ec19b03-kube-api-access-gdwf4\") pod \"maas-api-revoke-sub-b7c10330-94ldl\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:25.597109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.597079 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qpmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:27:25.597298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.597128 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 16:27:25.630307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:25.617377 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:26.040402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:26.039784 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl"] Nov 03 16:27:26.596472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:26.596433 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qpmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:27:26.596660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:26.596488 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 16:27:26.900021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:26.899987 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" event={"ID":"750a6935-778a-4b2f-b049-a9e07ec19b03","Type":"ContainerStarted","Data":"4ebc8d9f72c1f44ca6d1fcee82ff51f83733dd2c00f279a6a6a87dd64b58f9f3"} Nov 03 16:27:26.900184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:26.900048 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" event={"ID":"750a6935-778a-4b2f-b049-a9e07ec19b03","Type":"ContainerStarted","Data":"8e4c4e151fb808c681e6b0f7d120f8fbc7d2a8e46445d676918453097f5eff4b"} Nov 03 16:27:27.596734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:27.596694 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qpmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:27:27.597140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:27.596762 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 16:27:27.918448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:27.918339 5946 generic.go:356] "Generic (PLEG): container finished" podID="750a6935-778a-4b2f-b049-a9e07ec19b03" containerID="4ebc8d9f72c1f44ca6d1fcee82ff51f83733dd2c00f279a6a6a87dd64b58f9f3" exitCode=0 Nov 03 16:27:27.918694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:27.918661 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" event={"ID":"750a6935-778a-4b2f-b049-a9e07ec19b03","Type":"ContainerDied","Data":"4ebc8d9f72c1f44ca6d1fcee82ff51f83733dd2c00f279a6a6a87dd64b58f9f3"} Nov 03 16:27:27.941314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:27.941252 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" podStartSLOduration=2.9412330410000003 podStartE2EDuration="2.941233041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:27.068166618 +0000 UTC m=+917.189219442" watchObservedRunningTime="2026-11-03 16:27:27.941233041 +0000 UTC m=+918.062285868" Nov 03 16:27:27.942112 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:27.942091 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x"] Nov 03 16:27:27.945958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:27.945938 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.007068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.007035 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x"] Nov 03 16:27:28.028728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.028683 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c48378f-a619-4805-8685-7acec80579e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.029380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.029351 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvnw\" (UniqueName: \"kubernetes.io/projected/6c48378f-a619-4805-8685-7acec80579e8-kube-api-access-9kvnw\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.029525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.029483 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c48378f-a619-4805-8685-7acec80579e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.130498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.130458 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c48378f-a619-4805-8685-7acec80579e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.130674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.130563 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c48378f-a619-4805-8685-7acec80579e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.130674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.130637 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvnw\" (UniqueName: \"kubernetes.io/projected/6c48378f-a619-4805-8685-7acec80579e8-kube-api-access-9kvnw\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.131305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.131268 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c48378f-a619-4805-8685-7acec80579e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.133141 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.133110 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c48378f-a619-4805-8685-7acec80579e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.191381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.191311 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvnw\" (UniqueName: \"kubernetes.io/projected/6c48378f-a619-4805-8685-7acec80579e8-kube-api-access-9kvnw\") pod \"maas-api-revoke-sub-01a1c070-qxl6x\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.260429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.260384 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:28.450413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.450323 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:28.697278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.697246 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x"] Nov 03 16:27:28.698348 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:28.698321 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c48378f_a619_4805_8685_7acec80579e8.slice/crio-d9c7308ea5e0684c41dfa07ac91ae1a6babeb0fb74b5a91915f21d7afadd0b3f WatchSource:0}: Error finding container d9c7308ea5e0684c41dfa07ac91ae1a6babeb0fb74b5a91915f21d7afadd0b3f: Status 404 returned error can't find the container with id d9c7308ea5e0684c41dfa07ac91ae1a6babeb0fb74b5a91915f21d7afadd0b3f Nov 03 16:27:28.708268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.708240 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:28.708657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.708640 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:28.709486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.709468 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:27:28.926659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:28.926621 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" event={"ID":"6c48378f-a619-4805-8685-7acec80579e8","Type":"ContainerStarted","Data":"d9c7308ea5e0684c41dfa07ac91ae1a6babeb0fb74b5a91915f21d7afadd0b3f"} Nov 03 16:27:29.086297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.086273 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:29.146691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.146664 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750a6935-778a-4b2f-b049-a9e07ec19b03-kube-api-access-gdwf4\" (UniqueName: \"kubernetes.io/projected/750a6935-778a-4b2f-b049-a9e07ec19b03-kube-api-access-gdwf4\") pod \"750a6935-778a-4b2f-b049-a9e07ec19b03\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " Nov 03 16:27:29.146840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.146699 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-openshift-ca\") pod \"750a6935-778a-4b2f-b049-a9e07ec19b03\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " Nov 03 16:27:29.146840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.146754 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-xks-ca\") pod \"750a6935-778a-4b2f-b049-a9e07ec19b03\" (UID: \"750a6935-778a-4b2f-b049-a9e07ec19b03\") " Nov 03 16:27:29.147151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.147121 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-openshift-ca" pod "750a6935-778a-4b2f-b049-a9e07ec19b03" (UID: "750a6935-778a-4b2f-b049-a9e07ec19b03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:29.149018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.148993 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-xks-ca" pod "750a6935-778a-4b2f-b049-a9e07ec19b03" (UID: "750a6935-778a-4b2f-b049-a9e07ec19b03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:29.149122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.149097 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750a6935-778a-4b2f-b049-a9e07ec19b03-kube-api-access-gdwf4" pod "750a6935-778a-4b2f-b049-a9e07ec19b03" (UID: "750a6935-778a-4b2f-b049-a9e07ec19b03"). InnerVolumeSpecName "kube-api-access-gdwf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:29.167060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.167022 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:29.247914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.247834 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwf4\" (UniqueName: \"kubernetes.io/projected/750a6935-778a-4b2f-b049-a9e07ec19b03-kube-api-access-gdwf4\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:29.247914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.247871 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:29.247914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.247882 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a6935-778a-4b2f-b049-a9e07ec19b03-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:29.452662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.452631 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:29.938298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.938259 5946 generic.go:356] "Generic (PLEG): container finished" podID="6c48378f-a619-4805-8685-7acec80579e8" containerID="a660a2ee82413319f99d405214a93bde361e819196b3e433eae836dc263f7a00" exitCode=0 Nov 03 16:27:29.938839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.938357 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" event={"ID":"6c48378f-a619-4805-8685-7acec80579e8","Type":"ContainerDied","Data":"a660a2ee82413319f99d405214a93bde361e819196b3e433eae836dc263f7a00"} Nov 03 16:27:29.940166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.940136 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" Nov 03 16:27:29.940166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.940161 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl" event={"ID":"750a6935-778a-4b2f-b049-a9e07ec19b03","Type":"ContainerDied","Data":"8e4c4e151fb808c681e6b0f7d120f8fbc7d2a8e46445d676918453097f5eff4b"} Nov 03 16:27:29.940397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:29.940231 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e4c4e151fb808c681e6b0f7d120f8fbc7d2a8e46445d676918453097f5eff4b" Nov 03 16:27:30.765844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.765808 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq"] Nov 03 16:27:30.766643 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.766617 5946 cpu_manager.go:414] "containerMap: removing container" podUID="750a6935-778a-4b2f-b049-a9e07ec19b03" containerName="revoke-keys" Nov 03 16:27:30.766643 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.766641 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750a6935-778a-4b2f-b049-a9e07ec19b03" containerName="revoke-keys" Nov 03 16:27:30.766847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.766769 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="750a6935-778a-4b2f-b049-a9e07ec19b03" containerName="revoke-keys" Nov 03 16:27:30.770769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.770747 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.829401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.829368 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq"] Nov 03 16:27:30.869801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.869759 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.869950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.869845 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j68k7\" (UniqueName: \"kubernetes.io/projected/7725f188-5cf8-44bf-a684-d78b97aef0a4-kube-api-access-j68k7\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.870167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.870145 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.970918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.970878 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.971351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.970930 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.971351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.970956 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j68k7\" (UniqueName: \"kubernetes.io/projected/7725f188-5cf8-44bf-a684-d78b97aef0a4-kube-api-access-j68k7\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.972258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.972229 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:30.973575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:30.973556 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:31.031993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.031904 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j68k7\" (UniqueName: \"kubernetes.io/projected/7725f188-5cf8-44bf-a684-d78b97aef0a4-kube-api-access-j68k7\") pod \"maas-api-revoke-sub-5c520da9-vbzzq\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:31.085037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.085008 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:31.114582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.114558 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:31.173395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.173365 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c48378f-a619-4805-8685-7acec80579e8-kube-api-access-9kvnw\" (UniqueName: \"kubernetes.io/projected/6c48378f-a619-4805-8685-7acec80579e8-kube-api-access-9kvnw\") pod \"6c48378f-a619-4805-8685-7acec80579e8\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " Nov 03 16:27:31.173585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.173488 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c48378f-a619-4805-8685-7acec80579e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c48378f-a619-4805-8685-7acec80579e8-maas-api-openshift-ca\") pod \"6c48378f-a619-4805-8685-7acec80579e8\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " Nov 03 16:27:31.173660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.173600 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c48378f-a619-4805-8685-7acec80579e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c48378f-a619-4805-8685-7acec80579e8-maas-api-xks-ca\") pod \"6c48378f-a619-4805-8685-7acec80579e8\" (UID: \"6c48378f-a619-4805-8685-7acec80579e8\") " Nov 03 16:27:31.173839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.173818 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c48378f-a619-4805-8685-7acec80579e8-maas-api-openshift-ca" pod "6c48378f-a619-4805-8685-7acec80579e8" (UID: "6c48378f-a619-4805-8685-7acec80579e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:31.174126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.174104 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c48378f-a619-4805-8685-7acec80579e8-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:31.176368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.176338 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c48378f-a619-4805-8685-7acec80579e8-kube-api-access-9kvnw" pod "6c48378f-a619-4805-8685-7acec80579e8" (UID: "6c48378f-a619-4805-8685-7acec80579e8"). InnerVolumeSpecName "kube-api-access-9kvnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:31.176555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.176535 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c48378f-a619-4805-8685-7acec80579e8-maas-api-xks-ca" pod "6c48378f-a619-4805-8685-7acec80579e8" (UID: "6c48378f-a619-4805-8685-7acec80579e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:31.275057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.275016 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c48378f-a619-4805-8685-7acec80579e8-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:31.275057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.275046 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvnw\" (UniqueName: \"kubernetes.io/projected/6c48378f-a619-4805-8685-7acec80579e8-kube-api-access-9kvnw\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:31.469667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.469631 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq"] Nov 03 16:27:31.469878 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:31.469855 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7725f188_5cf8_44bf_a684_d78b97aef0a4.slice/crio-8cf6cefd4f8d0985b7554b8dc417156d0cb85acf7f371dec2f08ae225c6d00f9 WatchSource:0}: Error finding container 8cf6cefd4f8d0985b7554b8dc417156d0cb85acf7f371dec2f08ae225c6d00f9: Status 404 returned error can't find the container with id 8cf6cefd4f8d0985b7554b8dc417156d0cb85acf7f371dec2f08ae225c6d00f9 Nov 03 16:27:31.964129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.964097 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" Nov 03 16:27:31.964422 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.964124 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x" event={"ID":"6c48378f-a619-4805-8685-7acec80579e8","Type":"ContainerDied","Data":"d9c7308ea5e0684c41dfa07ac91ae1a6babeb0fb74b5a91915f21d7afadd0b3f"} Nov 03 16:27:31.964422 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.964185 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c7308ea5e0684c41dfa07ac91ae1a6babeb0fb74b5a91915f21d7afadd0b3f" Nov 03 16:27:31.966516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.966489 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" event={"ID":"7725f188-5cf8-44bf-a684-d78b97aef0a4","Type":"ContainerStarted","Data":"17df887a069e6ea600b8c05a6d90ac38487db509c3b2b4064cc77d397bef3879"} Nov 03 16:27:31.966662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.966537 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" event={"ID":"7725f188-5cf8-44bf-a684-d78b97aef0a4","Type":"ContainerStarted","Data":"8cf6cefd4f8d0985b7554b8dc417156d0cb85acf7f371dec2f08ae225c6d00f9"} Nov 03 16:27:31.973988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.973962 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs"] Nov 03 16:27:31.974608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.974590 5946 cpu_manager.go:414] "containerMap: removing container" podUID="6c48378f-a619-4805-8685-7acec80579e8" containerName="revoke-keys" Nov 03 16:27:31.974688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.974612 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c48378f-a619-4805-8685-7acec80579e8" containerName="revoke-keys" Nov 03 16:27:31.974782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.974769 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c48378f-a619-4805-8685-7acec80579e8" containerName="revoke-keys" Nov 03 16:27:31.978249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:31.978232 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.035581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.035551 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs"] Nov 03 16:27:32.083302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.083266 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.083460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.083366 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.083523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.083492 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhmp\" (UniqueName: \"kubernetes.io/projected/7cf26191-7e04-4187-9c38-f2f16ad202f8-kube-api-access-gzhmp\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.086199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.086154 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" podStartSLOduration=2.086142697 podStartE2EDuration="2.086142697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:32.085108366 +0000 UTC m=+922.206161188" watchObservedRunningTime="2026-11-03 16:27:32.086142697 +0000 UTC m=+922.207195566" Nov 03 16:27:32.184765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.184736 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.184931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.184784 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhmp\" (UniqueName: \"kubernetes.io/projected/7cf26191-7e04-4187-9c38-f2f16ad202f8-kube-api-access-gzhmp\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.184931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.184871 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.185575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.185550 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.187274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.187255 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.246861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.246798 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhmp\" (UniqueName: \"kubernetes.io/projected/7cf26191-7e04-4187-9c38-f2f16ad202f8-kube-api-access-gzhmp\") pod \"maas-api-revoke-sub-fbb00395-6gbqs\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.304516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.304485 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:32.684001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.683975 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs"] Nov 03 16:27:32.685069 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:32.685045 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf26191_7e04_4187_9c38_f2f16ad202f8.slice/crio-cbe6d8a9ab1b64758f5e709ad12e1d19ab56a4457b1457b1d62e03ebb2040f8b WatchSource:0}: Error finding container cbe6d8a9ab1b64758f5e709ad12e1d19ab56a4457b1457b1d62e03ebb2040f8b: Status 404 returned error can't find the container with id cbe6d8a9ab1b64758f5e709ad12e1d19ab56a4457b1457b1d62e03ebb2040f8b Nov 03 16:27:32.974643 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.974555 5946 generic.go:356] "Generic (PLEG): container finished" podID="7725f188-5cf8-44bf-a684-d78b97aef0a4" containerID="17df887a069e6ea600b8c05a6d90ac38487db509c3b2b4064cc77d397bef3879" exitCode=0 Nov 03 16:27:32.975091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.974643 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" event={"ID":"7725f188-5cf8-44bf-a684-d78b97aef0a4","Type":"ContainerDied","Data":"17df887a069e6ea600b8c05a6d90ac38487db509c3b2b4064cc77d397bef3879"} Nov 03 16:27:32.976190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:32.976156 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" event={"ID":"7cf26191-7e04-4187-9c38-f2f16ad202f8","Type":"ContainerStarted","Data":"cbe6d8a9ab1b64758f5e709ad12e1d19ab56a4457b1457b1d62e03ebb2040f8b"} Nov 03 16:27:33.087471 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:33.087440 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:33.983960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:33.983929 5946 generic.go:356] "Generic (PLEG): container finished" podID="7cf26191-7e04-4187-9c38-f2f16ad202f8" containerID="2ea4a09d1501f82682587604f2d60b506f66533fd8c14d05b10343e9d30f112d" exitCode=0 Nov 03 16:27:33.984406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:33.984015 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" event={"ID":"7cf26191-7e04-4187-9c38-f2f16ad202f8","Type":"ContainerDied","Data":"2ea4a09d1501f82682587604f2d60b506f66533fd8c14d05b10343e9d30f112d"} Nov 03 16:27:34.128907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.128885 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:34.202490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.202462 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-openshift-ca\") pod \"7725f188-5cf8-44bf-a684-d78b97aef0a4\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " Nov 03 16:27:34.202633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.202620 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7725f188-5cf8-44bf-a684-d78b97aef0a4-kube-api-access-j68k7\" (UniqueName: \"kubernetes.io/projected/7725f188-5cf8-44bf-a684-d78b97aef0a4-kube-api-access-j68k7\") pod \"7725f188-5cf8-44bf-a684-d78b97aef0a4\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " Nov 03 16:27:34.202693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.202676 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-xks-ca\") pod \"7725f188-5cf8-44bf-a684-d78b97aef0a4\" (UID: \"7725f188-5cf8-44bf-a684-d78b97aef0a4\") " Nov 03 16:27:34.202859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.202827 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-openshift-ca" pod "7725f188-5cf8-44bf-a684-d78b97aef0a4" (UID: "7725f188-5cf8-44bf-a684-d78b97aef0a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:34.203063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.203039 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:34.204977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.204954 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-xks-ca" pod "7725f188-5cf8-44bf-a684-d78b97aef0a4" (UID: "7725f188-5cf8-44bf-a684-d78b97aef0a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:34.205060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.204983 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7725f188-5cf8-44bf-a684-d78b97aef0a4-kube-api-access-j68k7" pod "7725f188-5cf8-44bf-a684-d78b97aef0a4" (UID: "7725f188-5cf8-44bf-a684-d78b97aef0a4"). InnerVolumeSpecName "kube-api-access-j68k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:34.304166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.304094 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j68k7\" (UniqueName: \"kubernetes.io/projected/7725f188-5cf8-44bf-a684-d78b97aef0a4-kube-api-access-j68k7\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:34.304166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.304122 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7725f188-5cf8-44bf-a684-d78b97aef0a4-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:34.395873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.395837 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg"] Nov 03 16:27:34.396359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.396345 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7725f188-5cf8-44bf-a684-d78b97aef0a4" containerName="revoke-keys" Nov 03 16:27:34.396420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.396360 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7725f188-5cf8-44bf-a684-d78b97aef0a4" containerName="revoke-keys" Nov 03 16:27:34.396462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.396457 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7725f188-5cf8-44bf-a684-d78b97aef0a4" containerName="revoke-keys" Nov 03 16:27:34.399887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.399872 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.469308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.469277 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg"] Nov 03 16:27:34.485022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.484992 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg"] Nov 03 16:27:34.488970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.488949 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.505751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.505730 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.505860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.505769 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.505860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.505793 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.505945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.505887 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6q92\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-kube-api-access-k6q92\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.505945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.505915 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.506020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.505945 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d0289-081e-4869-81fd-6c2a2b3621df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.506020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.505972 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.506088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.506018 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.506132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.506121 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.570228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.570125 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg"] Nov 03 16:27:34.607222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607184 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607259 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d0289-081e-4869-81fd-6c2a2b3621df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607290 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dvj\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-kube-api-access-z2dvj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607319 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607342 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607489 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607536 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607558 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607577 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607637 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607704 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607711 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f329-b09b-4d11-b70f-79b1835336ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607771 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607798 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.607903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607888 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d0289-081e-4869-81fd-6c2a2b3621df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607900 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.607968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607932 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.608027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.607995 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.608103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.608073 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.608171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.608123 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.608295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.608271 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6q92\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-kube-api-access-k6q92\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.608368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.608315 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.608436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.608379 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.610065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.610039 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.610155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.610144 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.668612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.668584 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.668911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.668889 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6q92\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-kube-api-access-k6q92\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ds4jg\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.709589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709551 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dvj\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-kube-api-access-z2dvj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.709745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709603 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.709745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709716 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.709874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709786 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.709874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709839 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.709973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709878 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f329-b09b-4d11-b70f-79b1835336ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.709973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709946 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.710078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709973 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.710078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.709994 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.710078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.710017 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.710319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.710293 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.710470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.710428 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.710550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.710491 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.710702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.710681 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f329-b09b-4d11-b70f-79b1835336ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.712257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.712233 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.712590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.712574 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.712739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.712728 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:34.770457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.770401 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.771022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.770997 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dvj\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-kube-api-access-z2dvj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2vqg\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.804599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.804569 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:34.992098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.992063 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" Nov 03 16:27:34.992606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.992095 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq" event={"ID":"7725f188-5cf8-44bf-a684-d78b97aef0a4","Type":"ContainerDied","Data":"8cf6cefd4f8d0985b7554b8dc417156d0cb85acf7f371dec2f08ae225c6d00f9"} Nov 03 16:27:34.992606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:34.992144 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cf6cefd4f8d0985b7554b8dc417156d0cb85acf7f371dec2f08ae225c6d00f9" Nov 03 16:27:35.141328 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.141300 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:35.146956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.146927 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg"] Nov 03 16:27:35.186770 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:35.186738 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6d0289_081e_4869_81fd_6c2a2b3621df.slice/crio-6516b33fb72e28d588d1833ee089c46278d50a4cac5c427c1544fdf1175513c1 WatchSource:0}: Error finding container 6516b33fb72e28d588d1833ee089c46278d50a4cac5c427c1544fdf1175513c1: Status 404 returned error can't find the container with id 6516b33fb72e28d588d1833ee089c46278d50a4cac5c427c1544fdf1175513c1 Nov 03 16:27:35.189300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.189259 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:35.189396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.189343 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:35.189396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.189388 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:35.203196 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.203168 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg"] Nov 03 16:27:35.203743 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:35.203720 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed1f329_b09b_4d11_b70f_79b1835336ed.slice/crio-0cd7aa72621b604cfa2052ee8847b9aa15bfc0a1ee6e8704da3a938e5ac13869 WatchSource:0}: Error finding container 0cd7aa72621b604cfa2052ee8847b9aa15bfc0a1ee6e8704da3a938e5ac13869: Status 404 returned error can't find the container with id 0cd7aa72621b604cfa2052ee8847b9aa15bfc0a1ee6e8704da3a938e5ac13869 Nov 03 16:27:35.206629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.206602 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:35.206692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.206663 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:35.206724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.206695 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:35.215242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.215217 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-xks-ca\") pod \"7cf26191-7e04-4187-9c38-f2f16ad202f8\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " Nov 03 16:27:35.215357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.215286 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-openshift-ca\") pod \"7cf26191-7e04-4187-9c38-f2f16ad202f8\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " Nov 03 16:27:35.215416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.215362 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf26191-7e04-4187-9c38-f2f16ad202f8-kube-api-access-gzhmp\" (UniqueName: \"kubernetes.io/projected/7cf26191-7e04-4187-9c38-f2f16ad202f8-kube-api-access-gzhmp\") pod \"7cf26191-7e04-4187-9c38-f2f16ad202f8\" (UID: \"7cf26191-7e04-4187-9c38-f2f16ad202f8\") " Nov 03 16:27:35.215638 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.215616 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-openshift-ca" pod "7cf26191-7e04-4187-9c38-f2f16ad202f8" (UID: "7cf26191-7e04-4187-9c38-f2f16ad202f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:35.217546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.217522 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf26191-7e04-4187-9c38-f2f16ad202f8-kube-api-access-gzhmp" pod "7cf26191-7e04-4187-9c38-f2f16ad202f8" (UID: "7cf26191-7e04-4187-9c38-f2f16ad202f8"). InnerVolumeSpecName "kube-api-access-gzhmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:35.217743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.217726 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-xks-ca" pod "7cf26191-7e04-4187-9c38-f2f16ad202f8" (UID: "7cf26191-7e04-4187-9c38-f2f16ad202f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:35.321913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.321877 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:35.321913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.321914 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf26191-7e04-4187-9c38-f2f16ad202f8-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:35.322159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:35.321929 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzhmp\" (UniqueName: \"kubernetes.io/projected/7cf26191-7e04-4187-9c38-f2f16ad202f8-kube-api-access-gzhmp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:36.003406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.003371 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" event={"ID":"9b6d0289-081e-4869-81fd-6c2a2b3621df","Type":"ContainerStarted","Data":"fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e"} Nov 03 16:27:36.003901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.003432 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" event={"ID":"9b6d0289-081e-4869-81fd-6c2a2b3621df","Type":"ContainerStarted","Data":"6516b33fb72e28d588d1833ee089c46278d50a4cac5c427c1544fdf1175513c1"} Nov 03 16:27:36.004634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.004609 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" event={"ID":"3ed1f329-b09b-4d11-b70f-79b1835336ed","Type":"ContainerStarted","Data":"3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b"} Nov 03 16:27:36.004696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.004652 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" event={"ID":"3ed1f329-b09b-4d11-b70f-79b1835336ed","Type":"ContainerStarted","Data":"0cd7aa72621b604cfa2052ee8847b9aa15bfc0a1ee6e8704da3a938e5ac13869"} Nov 03 16:27:36.008575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.006582 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" event={"ID":"7cf26191-7e04-4187-9c38-f2f16ad202f8","Type":"ContainerDied","Data":"cbe6d8a9ab1b64758f5e709ad12e1d19ab56a4457b1457b1d62e03ebb2040f8b"} Nov 03 16:27:36.008575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.006624 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe6d8a9ab1b64758f5e709ad12e1d19ab56a4457b1457b1d62e03ebb2040f8b" Nov 03 16:27:36.008575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.006676 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs" Nov 03 16:27:36.180937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.180883 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podStartSLOduration=2.180868779 podStartE2EDuration="2.180868779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:36.178368157 +0000 UTC m=+926.299421015" watchObservedRunningTime="2026-11-03 16:27:36.180868779 +0000 UTC m=+926.301921602" Nov 03 16:27:36.335291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.334702 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podStartSLOduration=2.334682986 podStartE2EDuration="2.334682986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:36.333750731 +0000 UTC m=+926.454803557" watchObservedRunningTime="2026-11-03 16:27:36.334682986 +0000 UTC m=+926.455735811" Nov 03 16:27:36.713017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.712931 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:36.741231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.736753 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ds4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:27:36.741231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.736819 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:27:36.805089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.804986 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:36.806421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.806400 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2vqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:27:36.806503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:36.806445 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:27:37.717232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:37.713402 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ds4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:27:37.717232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:37.713475 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:27:37.805624 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:37.805589 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2vqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:27:37.805789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:37.805641 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:27:38.080383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.080224 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:38.728587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.728534 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ds4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:27:38.729405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.729371 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:27:38.771092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.771012 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw"] Nov 03 16:27:38.771826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.771800 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7cf26191-7e04-4187-9c38-f2f16ad202f8" containerName="revoke-keys" Nov 03 16:27:38.771929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.771828 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf26191-7e04-4187-9c38-f2f16ad202f8" containerName="revoke-keys" Nov 03 16:27:38.771989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.771979 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf26191-7e04-4187-9c38-f2f16ad202f8" containerName="revoke-keys" Nov 03 16:27:38.777403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.777384 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:38.805070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.805041 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2vqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:27:38.805266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.805105 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:27:38.874182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.874152 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw"] Nov 03 16:27:38.874531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.874504 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:27:38.891989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.891894 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:27:38.891989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.891950 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:27:38.892317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.892292 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:27:38.895770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.895741 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62825154-7181-400a-bae3-5f893371bb43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:38.895981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.895962 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98slt\" (UniqueName: \"kubernetes.io/projected/62825154-7181-400a-bae3-5f893371bb43-kube-api-access-98slt\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:38.896115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.896103 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62825154-7181-400a-bae3-5f893371bb43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:38.997674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.997639 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98slt\" (UniqueName: \"kubernetes.io/projected/62825154-7181-400a-bae3-5f893371bb43-kube-api-access-98slt\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:38.997848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.997703 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62825154-7181-400a-bae3-5f893371bb43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:38.997848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.997762 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62825154-7181-400a-bae3-5f893371bb43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:38.998393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:38.998370 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62825154-7181-400a-bae3-5f893371bb43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:39.000389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.000365 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62825154-7181-400a-bae3-5f893371bb43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:39.065814 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.065634 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98slt\" (UniqueName: \"kubernetes.io/projected/62825154-7181-400a-bae3-5f893371bb43-kube-api-access-98slt\") pod \"maas-api-revoke-sub-5bdd9168-m55zw\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:39.091928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.091900 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:39.564395 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:39.562907 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62825154_7181_400a_bae3_5f893371bb43.slice/crio-1adbfbcb24f47133a2310402c3e5d5e061bee6fff2e4658752137dc207a2c4e3 WatchSource:0}: Error finding container 1adbfbcb24f47133a2310402c3e5d5e061bee6fff2e4658752137dc207a2c4e3: Status 404 returned error can't find the container with id 1adbfbcb24f47133a2310402c3e5d5e061bee6fff2e4658752137dc207a2c4e3 Nov 03 16:27:39.574222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.574175 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw"] Nov 03 16:27:39.717236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.713471 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ds4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:27:39.717236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.713533 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:27:39.805625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.805591 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2vqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:27:39.805993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:39.805638 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:27:40.084124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:40.084095 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" event={"ID":"62825154-7181-400a-bae3-5f893371bb43","Type":"ContainerStarted","Data":"6e46869a9baa6c8609be3c1ee341a8b1d03205898068cfde267389a89d8b4048"} Nov 03 16:27:40.084330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:40.084145 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" event={"ID":"62825154-7181-400a-bae3-5f893371bb43","Type":"ContainerStarted","Data":"1adbfbcb24f47133a2310402c3e5d5e061bee6fff2e4658752137dc207a2c4e3"} Nov 03 16:27:40.208886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:40.208768 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" podStartSLOduration=2.208754534 podStartE2EDuration="2.208754534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:40.206566935 +0000 UTC m=+930.327619762" watchObservedRunningTime="2026-11-03 16:27:40.208754534 +0000 UTC m=+930.329807397" Nov 03 16:27:40.734024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:40.713288 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ds4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:27:40.734024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:40.713345 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:27:40.804919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:40.804889 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2vqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:27:40.805152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:40.805129 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:27:41.093351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:41.093314 5946 generic.go:356] "Generic (PLEG): container finished" podID="62825154-7181-400a-bae3-5f893371bb43" containerID="6e46869a9baa6c8609be3c1ee341a8b1d03205898068cfde267389a89d8b4048" exitCode=0 Nov 03 16:27:41.093759 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:41.093410 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" event={"ID":"62825154-7181-400a-bae3-5f893371bb43","Type":"ContainerDied","Data":"6e46869a9baa6c8609be3c1ee341a8b1d03205898068cfde267389a89d8b4048"} Nov 03 16:27:41.713846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:41.713800 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ds4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:27:41.714050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:41.713866 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:27:41.805756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:41.805719 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2vqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:27:41.805907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:41.805797 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:27:42.276690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.276658 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:42.370711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.370622 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62825154-7181-400a-bae3-5f893371bb43-kube-api-access-98slt\" (UniqueName: \"kubernetes.io/projected/62825154-7181-400a-bae3-5f893371bb43-kube-api-access-98slt\") pod \"62825154-7181-400a-bae3-5f893371bb43\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " Nov 03 16:27:42.370711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.370677 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62825154-7181-400a-bae3-5f893371bb43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62825154-7181-400a-bae3-5f893371bb43-maas-api-openshift-ca\") pod \"62825154-7181-400a-bae3-5f893371bb43\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " Nov 03 16:27:42.370910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.370739 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62825154-7181-400a-bae3-5f893371bb43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62825154-7181-400a-bae3-5f893371bb43-maas-api-xks-ca\") pod \"62825154-7181-400a-bae3-5f893371bb43\" (UID: \"62825154-7181-400a-bae3-5f893371bb43\") " Nov 03 16:27:42.371177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.371138 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62825154-7181-400a-bae3-5f893371bb43-maas-api-openshift-ca" pod "62825154-7181-400a-bae3-5f893371bb43" (UID: "62825154-7181-400a-bae3-5f893371bb43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:42.372935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.372905 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62825154-7181-400a-bae3-5f893371bb43-maas-api-xks-ca" pod "62825154-7181-400a-bae3-5f893371bb43" (UID: "62825154-7181-400a-bae3-5f893371bb43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:42.373047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.373021 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62825154-7181-400a-bae3-5f893371bb43-kube-api-access-98slt" pod "62825154-7181-400a-bae3-5f893371bb43" (UID: "62825154-7181-400a-bae3-5f893371bb43"). InnerVolumeSpecName "kube-api-access-98slt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:42.472855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.472811 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98slt\" (UniqueName: \"kubernetes.io/projected/62825154-7181-400a-bae3-5f893371bb43-kube-api-access-98slt\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:42.472855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.472858 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62825154-7181-400a-bae3-5f893371bb43-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:42.473054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.472877 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62825154-7181-400a-bae3-5f893371bb43-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:42.586034 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:27:42.585990 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:27:42.713336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.713251 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ds4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:27:42.713336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.713315 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:27:42.805883 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.805846 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2vqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:27:42.806054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:42.805903 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:27:43.040762 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.040721 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p"] Nov 03 16:27:43.043598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.043565 5946 cpu_manager.go:414] "containerMap: removing container" podUID="62825154-7181-400a-bae3-5f893371bb43" containerName="revoke-keys" Nov 03 16:27:43.043598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.043596 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62825154-7181-400a-bae3-5f893371bb43" containerName="revoke-keys" Nov 03 16:27:43.043824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.043767 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="62825154-7181-400a-bae3-5f893371bb43" containerName="revoke-keys" Nov 03 16:27:43.048333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.048309 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.085711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.085677 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:43.086924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.086888 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.087235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087187 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.087406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087385 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.087568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087548 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.087704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087677 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.087838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087728 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.087908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087883 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.087971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087927 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.088037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.087974 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5pl\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-kube-api-access-dx5pl\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.145274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.126823 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" event={"ID":"62825154-7181-400a-bae3-5f893371bb43","Type":"ContainerDied","Data":"1adbfbcb24f47133a2310402c3e5d5e061bee6fff2e4658752137dc207a2c4e3"} Nov 03 16:27:43.145274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.126890 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1adbfbcb24f47133a2310402c3e5d5e061bee6fff2e4658752137dc207a2c4e3" Nov 03 16:27:43.145274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.126964 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw" Nov 03 16:27:43.156832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.156799 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2bb620-openshift-default-dockercfg-m62nq\"" Nov 03 16:27:43.163652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.163625 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p"] Nov 03 16:27:43.189761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.189719 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.189937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.189781 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.189937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.189829 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5pl\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-kube-api-access-dx5pl\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.189937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.189905 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.189943 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.189968 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.189995 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.190022 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.190058 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.190194 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.190274 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.190980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.190954 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.191261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.191239 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.191991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.191946 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.194340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.194314 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.194746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.194721 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.256518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.255913 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5pl\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-kube-api-access-dx5pl\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.259924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.259892 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.371566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.371461 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:43.848402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.848366 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:43.849822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.849799 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:43.850413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.850398 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:27:43.859952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.859923 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p"] Nov 03 16:27:43.861321 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:43.861296 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8062d5e7_e26d_4a4f_9309_dc05bf1551b3.slice/crio-64f351e7486c54314b879d3b7c1417eb870a3de83403e27dda70dae4bd767c2e WatchSource:0}: Error finding container 64f351e7486c54314b879d3b7c1417eb870a3de83403e27dda70dae4bd767c2e: Status 404 returned error can't find the container with id 64f351e7486c54314b879d3b7c1417eb870a3de83403e27dda70dae4bd767c2e Nov 03 16:27:43.863953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.863926 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:43.864029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.863987 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:43.864029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.864015 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:27:43.870785 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.870764 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:43.871085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.871068 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:43.872186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.872146 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:27:43.963489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:43.963461 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:27:44.140696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:44.140587 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" event={"ID":"8062d5e7-e26d-4a4f-9309-dc05bf1551b3","Type":"ContainerStarted","Data":"6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631"} Nov 03 16:27:44.140696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:44.140650 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" event={"ID":"8062d5e7-e26d-4a4f-9309-dc05bf1551b3","Type":"ContainerStarted","Data":"64f351e7486c54314b879d3b7c1417eb870a3de83403e27dda70dae4bd767c2e"} Nov 03 16:27:44.364268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:44.364186 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podStartSLOduration=2.3641666900000002 podStartE2EDuration="2.36416669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:44.362800112 +0000 UTC m=+934.483852935" watchObservedRunningTime="2026-11-03 16:27:44.36416669 +0000 UTC m=+934.485219515" Nov 03 16:27:44.372073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:44.372037 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:44.373820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:44.373792 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:27:44.373978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:44.373844 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:27:45.372918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:45.372821 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:27:45.373536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:45.373501 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:27:46.371813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:46.371777 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:27:46.371993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:46.371842 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:27:47.371798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.371766 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:27:47.372165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.371823 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:27:47.554075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.554043 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5"] Nov 03 16:27:47.558574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.558552 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.631843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.631756 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5"] Nov 03 16:27:47.668751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.668711 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:27:47.668751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.668714 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:27:47.668751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.668714 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:27:47.668995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.668826 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:27:47.706053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.706020 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.706235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.706080 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.706341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.706316 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkn9j\" (UniqueName: \"kubernetes.io/projected/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-kube-api-access-pkn9j\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.807993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.807958 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.808219 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.808070 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkn9j\" (UniqueName: \"kubernetes.io/projected/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-kube-api-access-pkn9j\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.808219 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.808120 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.808768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.808741 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.810707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.810677 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.869510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.869478 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkn9j\" (UniqueName: \"kubernetes.io/projected/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-kube-api-access-pkn9j\") pod \"maas-api-revoke-sub-d1c48fae-tllm5\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:47.874315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:47.874296 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:48.281739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:48.281713 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5"] Nov 03 16:27:48.282796 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:27:48.282770 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78f6eb7_6c6a_4fd9_9597_6b56b3361058.slice/crio-727f9f25a1f44ddb67807619a6a813c5425f91aa0acc37c537266b2580d91c1c WatchSource:0}: Error finding container 727f9f25a1f44ddb67807619a6a813c5425f91aa0acc37c537266b2580d91c1c: Status 404 returned error can't find the container with id 727f9f25a1f44ddb67807619a6a813c5425f91aa0acc37c537266b2580d91c1c Nov 03 16:27:48.372344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:48.372293 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:27:48.372944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:48.372893 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:27:48.412534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:48.412505 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:49.206300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:49.206251 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" event={"ID":"c78f6eb7-6c6a-4fd9-9597-6b56b3361058","Type":"ContainerStarted","Data":"0977a68122c8f4f4baa4a1296506f91428f0eef865e82ccfa63b851624d2e9c4"} Nov 03 16:27:49.206479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:49.206340 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" event={"ID":"c78f6eb7-6c6a-4fd9-9597-6b56b3361058","Type":"ContainerStarted","Data":"727f9f25a1f44ddb67807619a6a813c5425f91aa0acc37c537266b2580d91c1c"} Nov 03 16:27:49.335234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:49.330676 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" podStartSLOduration=2.3306582430000002 podStartE2EDuration="2.330658243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:49.329678379 +0000 UTC m=+939.450731206" watchObservedRunningTime="2026-11-03 16:27:49.330658243 +0000 UTC m=+939.451711067" Nov 03 16:27:49.372343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:49.372313 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:27:49.372501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:49.372364 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:27:50.214386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:50.214345 5946 generic.go:356] "Generic (PLEG): container finished" podID="c78f6eb7-6c6a-4fd9-9597-6b56b3361058" containerID="0977a68122c8f4f4baa4a1296506f91428f0eef865e82ccfa63b851624d2e9c4" exitCode=0 Nov 03 16:27:50.214598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:50.214428 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" event={"ID":"c78f6eb7-6c6a-4fd9-9597-6b56b3361058","Type":"ContainerDied","Data":"0977a68122c8f4f4baa4a1296506f91428f0eef865e82ccfa63b851624d2e9c4"} Nov 03 16:27:50.372023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:50.371988 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:27:50.372240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:50.372057 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:27:51.396830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.396805 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:51.472912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.472880 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:51.473201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.473173 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:51.474343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.474325 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:27:51.573688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.573651 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-openshift-ca\") pod \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " Nov 03 16:27:51.573888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.573704 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-xks-ca\") pod \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " Nov 03 16:27:51.573888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.573784 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-kube-api-access-pkn9j\" (UniqueName: \"kubernetes.io/projected/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-kube-api-access-pkn9j\") pod \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\" (UID: \"c78f6eb7-6c6a-4fd9-9597-6b56b3361058\") " Nov 03 16:27:51.574051 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.574026 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-openshift-ca" pod "c78f6eb7-6c6a-4fd9-9597-6b56b3361058" (UID: "c78f6eb7-6c6a-4fd9-9597-6b56b3361058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:27:51.574405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.574366 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:51.576014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.575990 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-xks-ca" pod "c78f6eb7-6c6a-4fd9-9597-6b56b3361058" (UID: "c78f6eb7-6c6a-4fd9-9597-6b56b3361058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:27:51.576113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.576019 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-kube-api-access-pkn9j" pod "c78f6eb7-6c6a-4fd9-9597-6b56b3361058" (UID: "c78f6eb7-6c6a-4fd9-9597-6b56b3361058"). InnerVolumeSpecName "kube-api-access-pkn9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:27:51.675667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.675620 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:51.675667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:51.675665 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkn9j\" (UniqueName: \"kubernetes.io/projected/c78f6eb7-6c6a-4fd9-9597-6b56b3361058-kube-api-access-pkn9j\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:27:52.234442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:52.234410 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" Nov 03 16:27:52.234589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:52.234446 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5" event={"ID":"c78f6eb7-6c6a-4fd9-9597-6b56b3361058","Type":"ContainerDied","Data":"727f9f25a1f44ddb67807619a6a813c5425f91aa0acc37c537266b2580d91c1c"} Nov 03 16:27:52.234589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:52.234497 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727f9f25a1f44ddb67807619a6a813c5425f91aa0acc37c537266b2580d91c1c" Nov 03 16:27:53.121469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:53.121431 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:57.855911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:27:57.855868 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:02.741838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:02.741798 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:07.472564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:07.472527 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:12.267654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.267621 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:12.463678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.463642 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:12.590445 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:28:12.590350 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:28:12.680170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.680134 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb"] Nov 03 16:28:12.680723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.680704 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c78f6eb7-6c6a-4fd9-9597-6b56b3361058" containerName="revoke-keys" Nov 03 16:28:12.680795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.680723 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78f6eb7-6c6a-4fd9-9597-6b56b3361058" containerName="revoke-keys" Nov 03 16:28:12.680833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.680808 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78f6eb7-6c6a-4fd9-9597-6b56b3361058" containerName="revoke-keys" Nov 03 16:28:12.685566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.685546 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.793373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.793327 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.793402 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45c172-3446-4838-92a4-aa1b5a29428f-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.793901 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.793965 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.794013 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.794225 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.794288 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2m8c\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-kube-api-access-k2m8c\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.794335 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.794457 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.802950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.801025 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb"] Nov 03 16:28:12.803394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.803015 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42da0f-openshift-default-dockercfg-l8c4p\"" Nov 03 16:28:12.896066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.895989 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896027 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896172 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45c172-3446-4838-92a4-aa1b5a29428f-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896245 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896297 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896339 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896367 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896417 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2m8c\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-kube-api-access-k2m8c\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896466 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896524 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896657 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896740 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896798 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.896917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.896894 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45c172-3446-4838-92a4-aa1b5a29428f-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.898712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.898693 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.898966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.898949 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.956883 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.956855 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2m8c\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-kube-api-access-k2m8c\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.956994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.956858 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:12.998745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:12.998246 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:13.437490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:13.437454 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb"] Nov 03 16:28:13.438486 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:28:13.438457 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f45c172_3446_4838_92a4_aa1b5a29428f.slice/crio-6b0dc9fe080e7c22dab01f7851a34e1b8e3694673b0d939d17e74dbd4b56659e WatchSource:0}: Error finding container 6b0dc9fe080e7c22dab01f7851a34e1b8e3694673b0d939d17e74dbd4b56659e: Status 404 returned error can't find the container with id 6b0dc9fe080e7c22dab01f7851a34e1b8e3694673b0d939d17e74dbd4b56659e Nov 03 16:28:13.441090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:13.441056 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:28:13.441175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:13.441124 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:28:13.441175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:13.441156 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:28:14.381217 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:14.381178 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" event={"ID":"8f45c172-3446-4838-92a4-aa1b5a29428f","Type":"ContainerStarted","Data":"b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22"} Nov 03 16:28:14.381398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:14.381254 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" event={"ID":"8f45c172-3446-4838-92a4-aa1b5a29428f","Type":"ContainerStarted","Data":"6b0dc9fe080e7c22dab01f7851a34e1b8e3694673b0d939d17e74dbd4b56659e"} Nov 03 16:28:14.565092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:14.565037 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" podStartSLOduration=2.565022169 podStartE2EDuration="2.565022169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:14.560898098 +0000 UTC m=+964.681950933" watchObservedRunningTime="2026-11-03 16:28:14.565022169 +0000 UTC m=+964.686075062" Nov 03 16:28:14.998777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:14.998741 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:14.999951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:14.999926 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:28:15.000065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:14.999981 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:28:15.999151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:15.999115 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:28:15.999565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:15.999168 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:28:16.999608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:16.999568 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:28:17.000050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:16.999639 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:28:17.260124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:17.260019 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:17.999415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:17.999376 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:28:17.999593 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:17.999437 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:28:18.999478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:18.999443 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:28:18.999866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:18.999494 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:28:20.126351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:20.126321 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:20.127069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:20.127047 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:20.127881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:20.127859 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:28:22.072033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:22.071985 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:24.004218 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:24.004175 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:28:25.970795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:25.970758 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5"] Nov 03 16:28:25.974842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:25.974826 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.074536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.074506 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-dshm\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.074700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.074547 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tmp-dir\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.074700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.074641 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-model-cache\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.074700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.074688 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-home\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.074828 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.074811 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tls-certs\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.074872 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.074859 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdxm2\" (UniqueName: \"kubernetes.io/projected/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-kube-api-access-mdxm2\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.083524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.083499 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"distinct-w1-bcd756-kserve-self-signed-certs\"" Nov 03 16:28:26.086932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.086907 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5"] Nov 03 16:28:26.105619 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.105575 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:26.176508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.176474 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-home\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.176681 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.176578 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tls-certs\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.176681 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.176631 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdxm2\" (UniqueName: \"kubernetes.io/projected/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-kube-api-access-mdxm2\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.176811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.176713 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-dshm\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.176811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.176752 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tmp-dir\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.176811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.176786 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-model-cache\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.176967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.176930 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-home\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.177101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.177073 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-model-cache\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.177672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.177645 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tmp-dir\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.179697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.179675 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tls-certs\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.179843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.179821 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-dshm\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.238610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.238572 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdxm2\" (UniqueName: \"kubernetes.io/projected/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-kube-api-access-mdxm2\") pod \"distinct-w1-bcd756-kserve-5fc9897569-k8lt5\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.287061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.287021 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:26.705757 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:28:26.705723 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fc48d03_da3f_4e61_9cf3_fadfaf0c27da.slice/crio-f7f7d86540d0e786c992e90c0133479a0caaf9a2ec2d831f0587245b0fd01271 WatchSource:0}: Error finding container f7f7d86540d0e786c992e90c0133479a0caaf9a2ec2d831f0587245b0fd01271: Status 404 returned error can't find the container with id f7f7d86540d0e786c992e90c0133479a0caaf9a2ec2d831f0587245b0fd01271 Nov 03 16:28:26.706715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.706681 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5"] Nov 03 16:28:26.879955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:26.879915 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:27.468787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:27.468744 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" event={"ID":"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da","Type":"ContainerStarted","Data":"7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643"} Nov 03 16:28:27.469179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:27.468798 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" event={"ID":"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da","Type":"ContainerStarted","Data":"f7f7d86540d0e786c992e90c0133479a0caaf9a2ec2d831f0587245b0fd01271"} Nov 03 16:28:27.591123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:27.591063 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podStartSLOduration=2.591048165 podStartE2EDuration="2.591048165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:27.589925266 +0000 UTC m=+977.710978088" watchObservedRunningTime="2026-11-03 16:28:27.591048165 +0000 UTC m=+977.712101006" Nov 03 16:28:28.287910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:28.287874 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:29.493480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:29.493443 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:31.573590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:31.573550 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:32.053769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:32.053738 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:36.287337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:36.287299 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:36.300249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:36.300224 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:36.377092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:36.377047 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:36.543855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:36.543778 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:28:39.806266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:39.806231 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:41.152375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:41.152340 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:42.585352 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:28:42.585317 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:28:45.960416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:45.960060 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:49.739948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.739917 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6"] Nov 03 16:28:49.744070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.744047 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.843383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.843351 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6"] Nov 03 16:28:49.850190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.850156 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-home\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.850190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.850184 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tls-certs\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.850451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.850352 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tmp-dir\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.850513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.850453 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5nt\" (UniqueName: \"kubernetes.io/projected/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-kube-api-access-rl5nt\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.850552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.850526 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-model-cache\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.850621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.850606 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-dshm\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.851439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.851424 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"distinct-w0-42da0f-kserve-self-signed-certs\"" Nov 03 16:28:49.859676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.859653 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:49.952259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952221 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5nt\" (UniqueName: \"kubernetes.io/projected/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-kube-api-access-rl5nt\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952278 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-model-cache\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952374 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-dshm\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952426 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-home\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952447 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tls-certs\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952536 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tmp-dir\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952784 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-model-cache\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952824 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-home\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.952952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.952929 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tmp-dir\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.954965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.954937 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-dshm\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:49.955110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:49.955093 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tls-certs\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:50.012976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.012908 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5nt\" (UniqueName: \"kubernetes.io/projected/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-kube-api-access-rl5nt\") pod \"distinct-w0-42da0f-kserve-569d787f6b-85jr6\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:50.060341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.060308 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:50.360955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.360871 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:28:50.500166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.500127 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6"] Nov 03 16:28:50.500797 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:28:50.500772 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1b7675_38d0_4487_b70b_8d4b67b5ba5a.slice/crio-c623f7237931a6ab4867ad9f5e08a59a65133919a7326a3834ff87104357cdbc WatchSource:0}: Error finding container c623f7237931a6ab4867ad9f5e08a59a65133919a7326a3834ff87104357cdbc: Status 404 returned error can't find the container with id c623f7237931a6ab4867ad9f5e08a59a65133919a7326a3834ff87104357cdbc Nov 03 16:28:50.629450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.629359 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" event={"ID":"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a","Type":"ContainerStarted","Data":"eb5d96c44051df5158a2661310925a61c31d91b748594429ca6d034aaf3a8b64"} Nov 03 16:28:50.629450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.629413 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" event={"ID":"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a","Type":"ContainerStarted","Data":"c623f7237931a6ab4867ad9f5e08a59a65133919a7326a3834ff87104357cdbc"} Nov 03 16:28:50.762176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.762122 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podStartSLOduration=1.7621047669999998 podStartE2EDuration="1.762104767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:50.761548521 +0000 UTC m=+1000.882601346" watchObservedRunningTime="2026-11-03 16:28:50.762104767 +0000 UTC m=+1000.883157590" Nov 03 16:28:50.877309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:50.877276 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:51.060457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:51.060427 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:28:55.678925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:55.678885 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:59.136025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:28:59.135993 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:00.061030 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:00.061002 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:29:00.073595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:00.073570 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:29:00.567295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:00.567252 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:00.718827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:00.718795 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:29:01.911367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:01.911334 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:05.495166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:05.495128 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:10.410435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:10.410403 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:12.587181 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:29:12.587143 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:29:14.497918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.497886 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj"] Nov 03 16:29:14.503715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.503691 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.562610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.562516 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c88d54-openshift-default-dockercfg-7w7bk\"" Nov 03 16:29:14.616128 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.616091 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj"] Nov 03 16:29:14.695174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.695132 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.695174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.695174 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.695455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.695194 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrns\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-kube-api-access-rcrns\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.695455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.695242 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.695455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.695396 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.695804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.695590 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.696132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.696104 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.696309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.696285 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.696398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.696346 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.797983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.797893 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.797983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.797931 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798002 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798035 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798099 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798137 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798183 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798232 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798256 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrns\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-kube-api-access-rcrns\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798556 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.798946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798922 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.799049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.798990 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.799143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.799125 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.799362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.799337 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.801233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.801171 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.801319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.801282 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.867396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.867363 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:14.868907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:14.868879 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrns\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-kube-api-access-rcrns\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:15.123715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.123391 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:15.547377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.547339 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:15.592233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.592182 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj"] Nov 03 16:29:15.627108 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:29:15.627081 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d7237e_1293_41c8_b5df_9a9fadfc8c71.slice/crio-e5daa8dab363a062ddd5d4633217c75f3bb5abb55952f1e92250b7be9ea95988 WatchSource:0}: Error finding container e5daa8dab363a062ddd5d4633217c75f3bb5abb55952f1e92250b7be9ea95988: Status 404 returned error can't find the container with id e5daa8dab363a062ddd5d4633217c75f3bb5abb55952f1e92250b7be9ea95988 Nov 03 16:29:15.629950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.629913 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:29:15.630056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.630006 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:29:15.630056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.630049 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:29:15.842144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.842054 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" event={"ID":"60d7237e-1293-41c8-b5df-9a9fadfc8c71","Type":"ContainerStarted","Data":"cedd638919e4873c0ba040bd016e2e5d2cd5ac0720ea3949d480265ebb0cdec5"} Nov 03 16:29:15.842144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.842115 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" event={"ID":"60d7237e-1293-41c8-b5df-9a9fadfc8c71","Type":"ContainerStarted","Data":"e5daa8dab363a062ddd5d4633217c75f3bb5abb55952f1e92250b7be9ea95988"} Nov 03 16:29:15.979809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:15.979752 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podStartSLOduration=1.9797357660000001 podStartE2EDuration="1.979735766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:29:15.976666959 +0000 UTC m=+1026.097719801" watchObservedRunningTime="2026-11-03 16:29:15.979735766 +0000 UTC m=+1026.100788588" Nov 03 16:29:16.125648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:16.125554 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:16.129609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:16.129571 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:29:16.129822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:16.129641 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:29:17.125230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:17.124628 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:29:17.125230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:17.124693 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:29:18.125697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:18.125656 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:29:18.126744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:18.126322 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:29:19.135232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:19.130621 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:29:19.135232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:19.130686 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:29:20.134599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:20.134562 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:29:20.134888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:20.134858 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:29:20.690144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:20.690111 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:21.125643 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:21.125605 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:29:21.125954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:21.125921 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:29:22.124600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:22.124559 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:29:22.125126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:22.124627 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:29:22.652360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:22.652324 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:23.128477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:23.128450 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:23.128924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:23.128910 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:23.129627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:23.129611 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:29:25.599855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:25.599813 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:29.873160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:29.873105 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:30.465755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:30.465719 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:35.285187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:35.285150 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:38.459493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:38.459458 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:29:40.265595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:40.265547 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:42.585952 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:29:42.585911 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:29:45.068690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:45.068654 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:49.836419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:49.836380 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:54.776916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:54.776862 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:59.552449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:29:59.552416 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:00.187933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.187891 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847"] Nov 03 16:30:00.192687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.192662 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk"] Nov 03 16:30:00.192841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.192770 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.196943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.196919 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb"] Nov 03 16:30:00.197080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.197037 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.201518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.201497 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.239039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.239010 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn"] Nov 03 16:30:00.243662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.243642 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d"] Nov 03 16:30:00.243774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.243760 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.248899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.248873 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.249766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.249748 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:30:00.249896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.249785 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:30:00.249896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.249785 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:30:00.269640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.269617 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hjp\" (UniqueName: \"kubernetes.io/projected/c681a491-07ff-42e3-a4ac-29941b1c4172-kube-api-access-76hjp\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.269756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.269655 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c681a491-07ff-42e3-a4ac-29941b1c4172-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.303609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.303580 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk"] Nov 03 16:30:00.310428 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.310384 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn"] Nov 03 16:30:00.318117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.318084 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb"] Nov 03 16:30:00.344668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.344638 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d"] Nov 03 16:30:00.370995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.370929 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hjp\" (UniqueName: \"kubernetes.io/projected/c681a491-07ff-42e3-a4ac-29941b1c4172-kube-api-access-76hjp\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.371180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371002 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gzw\" (UniqueName: \"kubernetes.io/projected/04a4104f-4138-4b9d-894f-68c9d2605534-kube-api-access-42gzw\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.371180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371042 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vgn6\" (UniqueName: \"kubernetes.io/projected/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-kube-api-access-5vgn6\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.371180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371072 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c681a491-07ff-42e3-a4ac-29941b1c4172-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.371180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371145 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.371442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371222 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482258f-d167-4b50-9d1c-6b1c48073fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.371442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371315 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c9d64-b1d0-41e7-8589-30600f34c428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.371442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371411 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69f9\" (UniqueName: \"kubernetes.io/projected/3482258f-d167-4b50-9d1c-6b1c48073fee-kube-api-access-j69f9\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.371566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371507 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a4104f-4138-4b9d-894f-68c9d2605534-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.371604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371586 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8hz2\" (UniqueName: \"kubernetes.io/projected/969c9d64-b1d0-41e7-8589-30600f34c428-kube-api-access-l8hz2\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.371930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.371910 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c681a491-07ff-42e3-a4ac-29941b1c4172-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.407721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.407683 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847"] Nov 03 16:30:00.435537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.435486 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hjp\" (UniqueName: \"kubernetes.io/projected/c681a491-07ff-42e3-a4ac-29941b1c4172-kube-api-access-76hjp\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.473268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473168 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c9d64-b1d0-41e7-8589-30600f34c428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.473268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473240 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69f9\" (UniqueName: \"kubernetes.io/projected/3482258f-d167-4b50-9d1c-6b1c48073fee-kube-api-access-j69f9\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.473498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473315 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a4104f-4138-4b9d-894f-68c9d2605534-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.473498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473379 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8hz2\" (UniqueName: \"kubernetes.io/projected/969c9d64-b1d0-41e7-8589-30600f34c428-kube-api-access-l8hz2\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.473608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473557 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gzw\" (UniqueName: \"kubernetes.io/projected/04a4104f-4138-4b9d-894f-68c9d2605534-kube-api-access-42gzw\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.473608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473596 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vgn6\" (UniqueName: \"kubernetes.io/projected/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-kube-api-access-5vgn6\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.473713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473626 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.473713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473667 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482258f-d167-4b50-9d1c-6b1c48073fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.473963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.473940 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c9d64-b1d0-41e7-8589-30600f34c428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.474050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.474029 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a4104f-4138-4b9d-894f-68c9d2605534-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.474359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.474338 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482258f-d167-4b50-9d1c-6b1c48073fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.474453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.474429 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.509007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.508977 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:00.534537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.534507 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vgn6\" (UniqueName: \"kubernetes.io/projected/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-kube-api-access-5vgn6\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.535054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.535032 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gzw\" (UniqueName: \"kubernetes.io/projected/04a4104f-4138-4b9d-894f-68c9d2605534-kube-api-access-42gzw\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.535299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.535279 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69f9\" (UniqueName: \"kubernetes.io/projected/3482258f-d167-4b50-9d1c-6b1c48073fee-kube-api-access-j69f9\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.536798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.536777 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8hz2\" (UniqueName: \"kubernetes.io/projected/969c9d64-b1d0-41e7-8589-30600f34c428-kube-api-access-l8hz2\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.559889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.559860 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:00.565304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.565277 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:00.816968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.816939 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:00.828252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.828220 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:00.915629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.915549 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847"] Nov 03 16:30:00.922843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.922813 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk"] Nov 03 16:30:00.931905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.931885 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:00.977832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.977805 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk"] Nov 03 16:30:00.983035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.983007 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn"] Nov 03 16:30:00.987804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.987783 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"default-dockercfg-rwp4d\"" Nov 03 16:30:00.988421 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:30:00.988255 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3482258f_d167_4b50_9d1c_6b1c48073fee.slice/crio-6a550940342211c330703ccb29a7efa39829af4def5e8cfb9a29944e9c115be0 WatchSource:0}: Error finding container 6a550940342211c330703ccb29a7efa39829af4def5e8cfb9a29944e9c115be0: Status 404 returned error can't find the container with id 6a550940342211c330703ccb29a7efa39829af4def5e8cfb9a29944e9c115be0 Nov 03 16:30:00.988421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.988374 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"distinct-2-w4-2bb620-kserve-self-signed-certs\"" Nov 03 16:30:00.990184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:00.990167 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d"] Nov 03 16:30:01.018355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.018321 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"openshift-service-ca.crt\"" Nov 03 16:30:01.018537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.018519 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"kube-root-ca.crt\"" Nov 03 16:30:01.037800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.037748 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:01.085019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.084927 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-model-cache\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.085019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.084971 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-dshm\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.085288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.085051 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091c43-f73f-4b47-b221-9567bb1670ee-tls-certs\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.085288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.085233 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-tmp-dir\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.085433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.085414 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkfp\" (UniqueName: \"kubernetes.io/projected/40091c43-f73f-4b47-b221-9567bb1670ee-kube-api-access-2jkfp\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.085521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.085507 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-home\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.151080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.151036 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" event={"ID":"3482258f-d167-4b50-9d1c-6b1c48073fee","Type":"ContainerStarted","Data":"6a550940342211c330703ccb29a7efa39829af4def5e8cfb9a29944e9c115be0"} Nov 03 16:30:01.152498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.152471 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" event={"ID":"04a4104f-4138-4b9d-894f-68c9d2605534","Type":"ContainerStarted","Data":"c03a0b12635d53b3a58d880019d3d6ab305aa57cc215b6d6405b6ca35aad20b9"} Nov 03 16:30:01.153722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.153699 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" event={"ID":"c681a491-07ff-42e3-a4ac-29941b1c4172","Type":"ContainerStarted","Data":"0b6fd012e3bfb9f52ead028c624740abe02ad2a932e000808eb52763680bfb0e"} Nov 03 16:30:01.186354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186322 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkfp\" (UniqueName: \"kubernetes.io/projected/40091c43-f73f-4b47-b221-9567bb1670ee-kube-api-access-2jkfp\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.186513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186409 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-home\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.186513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186471 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-model-cache\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.186513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186503 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-dshm\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.186684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186559 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091c43-f73f-4b47-b221-9567bb1670ee-tls-certs\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.186684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186631 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-tmp-dir\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.186898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186872 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-home\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.186898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.186889 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-model-cache\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.187026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.187008 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-tmp-dir\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.189584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.189552 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-dshm\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.189711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.189693 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091c43-f73f-4b47-b221-9567bb1670ee-tls-certs\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.208811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.208779 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk"] Nov 03 16:30:01.209505 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:30:01.209475 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa19ddd_04a8_4e22_b539_8b6b913cbaf8.slice/crio-55c3d2ad53bfc1fba6fdba9e304128ca54d31235957ab505025674ff58f632d2 WatchSource:0}: Error finding container 55c3d2ad53bfc1fba6fdba9e304128ca54d31235957ab505025674ff58f632d2: Status 404 returned error can't find the container with id 55c3d2ad53bfc1fba6fdba9e304128ca54d31235957ab505025674ff58f632d2 Nov 03 16:30:01.223655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.223629 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb"] Nov 03 16:30:01.231921 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:30:01.231860 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969c9d64_b1d0_41e7_8589_30600f34c428.slice/crio-f7e3905f59b162f577a50e2f683a0a58fd5674e3b3db6309f375ce168351718c WatchSource:0}: Error finding container f7e3905f59b162f577a50e2f683a0a58fd5674e3b3db6309f375ce168351718c: Status 404 returned error can't find the container with id f7e3905f59b162f577a50e2f683a0a58fd5674e3b3db6309f375ce168351718c Nov 03 16:30:01.247303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.247277 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkfp\" (UniqueName: \"kubernetes.io/projected/40091c43-f73f-4b47-b221-9567bb1670ee-kube-api-access-2jkfp\") pod \"distinct-2-w4-2bb620-kserve-58b85c9996-jpntk\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.284036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.283689 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:01.804160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:01.804123 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk"] Nov 03 16:30:01.804956 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:30:01.804931 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40091c43_f73f_4b47_b221_9567bb1670ee.slice/crio-abe95090e88a52891aaa118788e725bdf8e6b4fbd520ced0292c0afa5408fb6e WatchSource:0}: Error finding container abe95090e88a52891aaa118788e725bdf8e6b4fbd520ced0292c0afa5408fb6e: Status 404 returned error can't find the container with id abe95090e88a52891aaa118788e725bdf8e6b4fbd520ced0292c0afa5408fb6e Nov 03 16:30:02.162600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.162567 5946 generic.go:356] "Generic (PLEG): container finished" podID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerID="aa2a9992fe055956e5139d3d551346906571b24c93dcd795633a1eb820ba8930" exitCode=60 Nov 03 16:30:02.162743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.162654 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" event={"ID":"3482258f-d167-4b50-9d1c-6b1c48073fee","Type":"ContainerDied","Data":"aa2a9992fe055956e5139d3d551346906571b24c93dcd795633a1eb820ba8930"} Nov 03 16:30:02.162928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.162912 5946 scope.go:122] "RemoveContainer" containerID="aa2a9992fe055956e5139d3d551346906571b24c93dcd795633a1eb820ba8930" Nov 03 16:30:02.164599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.164458 5946 generic.go:356] "Generic (PLEG): container finished" podID="04a4104f-4138-4b9d-894f-68c9d2605534" containerID="575b109b88bae64b75895d55d62f8ac07676764a851696839fc951cf37c197ca" exitCode=60 Nov 03 16:30:02.164599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.164550 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" event={"ID":"04a4104f-4138-4b9d-894f-68c9d2605534","Type":"ContainerDied","Data":"575b109b88bae64b75895d55d62f8ac07676764a851696839fc951cf37c197ca"} Nov 03 16:30:02.164769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.164748 5946 scope.go:122] "RemoveContainer" containerID="575b109b88bae64b75895d55d62f8ac07676764a851696839fc951cf37c197ca" Nov 03 16:30:02.166468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.166360 5946 generic.go:356] "Generic (PLEG): container finished" podID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerID="400192b172a66a7de483108fc4398417f9f2f657149defaddec383b95e9bb81b" exitCode=60 Nov 03 16:30:02.166617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.166592 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" event={"ID":"caa19ddd-04a8-4e22-b539-8b6b913cbaf8","Type":"ContainerDied","Data":"400192b172a66a7de483108fc4398417f9f2f657149defaddec383b95e9bb81b"} Nov 03 16:30:02.166719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.166641 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" event={"ID":"caa19ddd-04a8-4e22-b539-8b6b913cbaf8","Type":"ContainerStarted","Data":"55c3d2ad53bfc1fba6fdba9e304128ca54d31235957ab505025674ff58f632d2"} Nov 03 16:30:02.166719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.166698 5946 scope.go:122] "RemoveContainer" containerID="400192b172a66a7de483108fc4398417f9f2f657149defaddec383b95e9bb81b" Nov 03 16:30:02.168095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.168074 5946 generic.go:356] "Generic (PLEG): container finished" podID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerID="a3e6fe6d22239d00bf8993abfd7391a5226e850a0cf9bb052b155610719b4903" exitCode=60 Nov 03 16:30:02.168189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.168136 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" event={"ID":"c681a491-07ff-42e3-a4ac-29941b1c4172","Type":"ContainerDied","Data":"a3e6fe6d22239d00bf8993abfd7391a5226e850a0cf9bb052b155610719b4903"} Nov 03 16:30:02.168364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.168348 5946 scope.go:122] "RemoveContainer" containerID="a3e6fe6d22239d00bf8993abfd7391a5226e850a0cf9bb052b155610719b4903" Nov 03 16:30:02.169803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.169782 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" event={"ID":"969c9d64-b1d0-41e7-8589-30600f34c428","Type":"ContainerStarted","Data":"f2d7cab750eaa5bc9b81f661a5a742aa9e63d212f02381de4495128bd6fd9ce8"} Nov 03 16:30:02.169878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.169817 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" event={"ID":"969c9d64-b1d0-41e7-8589-30600f34c428","Type":"ContainerStarted","Data":"f7e3905f59b162f577a50e2f683a0a58fd5674e3b3db6309f375ce168351718c"} Nov 03 16:30:02.171247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.171227 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" event={"ID":"40091c43-f73f-4b47-b221-9567bb1670ee","Type":"ContainerStarted","Data":"47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6"} Nov 03 16:30:02.171338 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.171259 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" event={"ID":"40091c43-f73f-4b47-b221-9567bb1670ee","Type":"ContainerStarted","Data":"abe95090e88a52891aaa118788e725bdf8e6b4fbd520ced0292c0afa5408fb6e"} Nov 03 16:30:02.285122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.285086 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:02.645403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.645339 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" podStartSLOduration=2.645320832 podStartE2EDuration="2.645320832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:02.640763893 +0000 UTC m=+1072.761816728" watchObservedRunningTime="2026-11-03 16:30:02.645320832 +0000 UTC m=+1072.766373655" Nov 03 16:30:02.880537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:02.880478 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podStartSLOduration=2.880463053 podStartE2EDuration="2.880463053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:02.879408714 +0000 UTC m=+1073.000461549" watchObservedRunningTime="2026-11-03 16:30:02.880463053 +0000 UTC m=+1073.001515929" Nov 03 16:30:03.180949 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.180901 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" event={"ID":"caa19ddd-04a8-4e22-b539-8b6b913cbaf8","Type":"ContainerStarted","Data":"4d13199bcfdc0fa2952ebb8c7eb1e7c3044dc8c1a2cbe3d1a9bfd2c814c6a99d"} Nov 03 16:30:03.183176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.183148 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" event={"ID":"c681a491-07ff-42e3-a4ac-29941b1c4172","Type":"ContainerStarted","Data":"43096bb3099b9fe2f3d5545afca8c217f5d9b100ee3c08b46bb11c58ff5c6662"} Nov 03 16:30:03.185656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.185620 5946 generic.go:356] "Generic (PLEG): container finished" podID="969c9d64-b1d0-41e7-8589-30600f34c428" containerID="f2d7cab750eaa5bc9b81f661a5a742aa9e63d212f02381de4495128bd6fd9ce8" exitCode=60 Nov 03 16:30:03.185795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.185696 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" event={"ID":"969c9d64-b1d0-41e7-8589-30600f34c428","Type":"ContainerDied","Data":"f2d7cab750eaa5bc9b81f661a5a742aa9e63d212f02381de4495128bd6fd9ce8"} Nov 03 16:30:03.185973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.185957 5946 scope.go:122] "RemoveContainer" containerID="f2d7cab750eaa5bc9b81f661a5a742aa9e63d212f02381de4495128bd6fd9ce8" Nov 03 16:30:03.188296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.188273 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" event={"ID":"3482258f-d167-4b50-9d1c-6b1c48073fee","Type":"ContainerStarted","Data":"4ccc5dfb75b8747fa1a5cd38d8e2d6d24f39fac832c7dbec77e2178635b69c74"} Nov 03 16:30:03.190565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.190541 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" event={"ID":"04a4104f-4138-4b9d-894f-68c9d2605534","Type":"ContainerStarted","Data":"11f8f5dc4edf4a6b84c6fcdac1870e1bceb38e230b78537406eb50f4a99f0d88"} Nov 03 16:30:03.307911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.307857 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" podStartSLOduration=3.307843508 podStartE2EDuration="3.307843508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:03.305920813 +0000 UTC m=+1073.426973637" watchObservedRunningTime="2026-11-03 16:30:03.307843508 +0000 UTC m=+1073.428896330" Nov 03 16:30:03.427337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.427277 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" podStartSLOduration=3.427257642 podStartE2EDuration="3.427257642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:03.424241241 +0000 UTC m=+1073.545294064" watchObservedRunningTime="2026-11-03 16:30:03.427257642 +0000 UTC m=+1073.548310469" Nov 03 16:30:03.547077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.547018 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" podStartSLOduration=3.547004648 podStartE2EDuration="3.547004648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:03.543516991 +0000 UTC m=+1073.664569815" watchObservedRunningTime="2026-11-03 16:30:03.547004648 +0000 UTC m=+1073.668057471" Nov 03 16:30:03.781605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:03.781505 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" podStartSLOduration=3.781490312 podStartE2EDuration="3.781490312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:03.779953972 +0000 UTC m=+1073.901006795" watchObservedRunningTime="2026-11-03 16:30:03.781490312 +0000 UTC m=+1073.902543134" Nov 03 16:30:04.200984 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.200950 5946 generic.go:356] "Generic (PLEG): container finished" podID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerID="4ccc5dfb75b8747fa1a5cd38d8e2d6d24f39fac832c7dbec77e2178635b69c74" exitCode=60 Nov 03 16:30:04.201500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.201050 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" event={"ID":"3482258f-d167-4b50-9d1c-6b1c48073fee","Type":"ContainerDied","Data":"4ccc5dfb75b8747fa1a5cd38d8e2d6d24f39fac832c7dbec77e2178635b69c74"} Nov 03 16:30:04.201500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.201113 5946 scope.go:122] "RemoveContainer" containerID="aa2a9992fe055956e5139d3d551346906571b24c93dcd795633a1eb820ba8930" Nov 03 16:30:04.201500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.201350 5946 scope.go:122] "RemoveContainer" containerID="4ccc5dfb75b8747fa1a5cd38d8e2d6d24f39fac832c7dbec77e2178635b69c74" Nov 03 16:30:04.201714 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:04.201693 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d_odh-ai-gateway-infra(3482258f-d167-4b50-9d1c-6b1c48073fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" Nov 03 16:30:04.203175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.203155 5946 generic.go:356] "Generic (PLEG): container finished" podID="04a4104f-4138-4b9d-894f-68c9d2605534" containerID="11f8f5dc4edf4a6b84c6fcdac1870e1bceb38e230b78537406eb50f4a99f0d88" exitCode=60 Nov 03 16:30:04.203295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.203248 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" event={"ID":"04a4104f-4138-4b9d-894f-68c9d2605534","Type":"ContainerDied","Data":"11f8f5dc4edf4a6b84c6fcdac1870e1bceb38e230b78537406eb50f4a99f0d88"} Nov 03 16:30:04.203533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.203512 5946 scope.go:122] "RemoveContainer" containerID="11f8f5dc4edf4a6b84c6fcdac1870e1bceb38e230b78537406eb50f4a99f0d88" Nov 03 16:30:04.203812 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:04.203789 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn_odh-ai-gateway-infra(04a4104f-4138-4b9d-894f-68c9d2605534)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" Nov 03 16:30:04.205313 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.205294 5946 generic.go:356] "Generic (PLEG): container finished" podID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerID="4d13199bcfdc0fa2952ebb8c7eb1e7c3044dc8c1a2cbe3d1a9bfd2c814c6a99d" exitCode=60 Nov 03 16:30:04.205437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.205363 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" event={"ID":"caa19ddd-04a8-4e22-b539-8b6b913cbaf8","Type":"ContainerDied","Data":"4d13199bcfdc0fa2952ebb8c7eb1e7c3044dc8c1a2cbe3d1a9bfd2c814c6a99d"} Nov 03 16:30:04.205745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.205621 5946 scope.go:122] "RemoveContainer" containerID="4d13199bcfdc0fa2952ebb8c7eb1e7c3044dc8c1a2cbe3d1a9bfd2c814c6a99d" Nov 03 16:30:04.206110 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:04.206084 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk_odh-ai-gateway-infra(caa19ddd-04a8-4e22-b539-8b6b913cbaf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" Nov 03 16:30:04.207384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.207362 5946 generic.go:356] "Generic (PLEG): container finished" podID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerID="43096bb3099b9fe2f3d5545afca8c217f5d9b100ee3c08b46bb11c58ff5c6662" exitCode=60 Nov 03 16:30:04.207468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.207401 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" event={"ID":"c681a491-07ff-42e3-a4ac-29941b1c4172","Type":"ContainerDied","Data":"43096bb3099b9fe2f3d5545afca8c217f5d9b100ee3c08b46bb11c58ff5c6662"} Nov 03 16:30:04.207637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.207620 5946 scope.go:122] "RemoveContainer" containerID="43096bb3099b9fe2f3d5545afca8c217f5d9b100ee3c08b46bb11c58ff5c6662" Nov 03 16:30:04.207881 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:04.207861 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847_odh-ai-gateway-infra(c681a491-07ff-42e3-a4ac-29941b1c4172)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" Nov 03 16:30:04.209572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.209550 5946 generic.go:356] "Generic (PLEG): container finished" podID="969c9d64-b1d0-41e7-8589-30600f34c428" containerID="30e4feaec0d05bfbf0e514d529e38bc9554210118486bf6a197075763ff58bc7" exitCode=60 Nov 03 16:30:04.209673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.209635 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" event={"ID":"969c9d64-b1d0-41e7-8589-30600f34c428","Type":"ContainerDied","Data":"30e4feaec0d05bfbf0e514d529e38bc9554210118486bf6a197075763ff58bc7"} Nov 03 16:30:04.209835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.209817 5946 scope.go:122] "RemoveContainer" containerID="30e4feaec0d05bfbf0e514d529e38bc9554210118486bf6a197075763ff58bc7" Nov 03 16:30:04.210066 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:04.210041 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb_odh-ai-gateway-infra(969c9d64-b1d0-41e7-8589-30600f34c428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" Nov 03 16:30:04.215373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.215354 5946 scope.go:122] "RemoveContainer" containerID="575b109b88bae64b75895d55d62f8ac07676764a851696839fc951cf37c197ca" Nov 03 16:30:04.260720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.260697 5946 scope.go:122] "RemoveContainer" containerID="400192b172a66a7de483108fc4398417f9f2f657149defaddec383b95e9bb81b" Nov 03 16:30:04.275322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.275291 5946 scope.go:122] "RemoveContainer" containerID="a3e6fe6d22239d00bf8993abfd7391a5226e850a0cf9bb052b155610719b4903" Nov 03 16:30:04.288932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.288908 5946 scope.go:122] "RemoveContainer" containerID="f2d7cab750eaa5bc9b81f661a5a742aa9e63d212f02381de4495128bd6fd9ce8" Nov 03 16:30:04.364726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:04.364679 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:05.219490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:05.219456 5946 scope.go:122] "RemoveContainer" containerID="4d13199bcfdc0fa2952ebb8c7eb1e7c3044dc8c1a2cbe3d1a9bfd2c814c6a99d" Nov 03 16:30:05.220020 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:05.219829 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk_odh-ai-gateway-infra(caa19ddd-04a8-4e22-b539-8b6b913cbaf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" Nov 03 16:30:05.221181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:05.221164 5946 scope.go:122] "RemoveContainer" containerID="43096bb3099b9fe2f3d5545afca8c217f5d9b100ee3c08b46bb11c58ff5c6662" Nov 03 16:30:05.221480 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:05.221439 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847_odh-ai-gateway-infra(c681a491-07ff-42e3-a4ac-29941b1c4172)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" Nov 03 16:30:05.222798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:05.222783 5946 scope.go:122] "RemoveContainer" containerID="30e4feaec0d05bfbf0e514d529e38bc9554210118486bf6a197075763ff58bc7" Nov 03 16:30:05.223094 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:05.223057 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb_odh-ai-gateway-infra(969c9d64-b1d0-41e7-8589-30600f34c428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" Nov 03 16:30:05.224388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:05.224372 5946 scope.go:122] "RemoveContainer" containerID="4ccc5dfb75b8747fa1a5cd38d8e2d6d24f39fac832c7dbec77e2178635b69c74" Nov 03 16:30:05.224617 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:05.224585 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d_odh-ai-gateway-infra(3482258f-d167-4b50-9d1c-6b1c48073fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" Nov 03 16:30:05.226003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:05.225987 5946 scope.go:122] "RemoveContainer" containerID="11f8f5dc4edf4a6b84c6fcdac1870e1bceb38e230b78537406eb50f4a99f0d88" Nov 03 16:30:05.226237 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:05.226194 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn_odh-ai-gateway-infra(04a4104f-4138-4b9d-894f-68c9d2605534)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" Nov 03 16:30:09.075244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:09.074439 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:11.284804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:11.284770 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:11.297763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:11.297737 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:12.287717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:12.287684 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:30:12.591904 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:12.591824 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:30:13.880820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:13.880744 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:15.997393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:15.997353 5946 scope.go:122] "RemoveContainer" containerID="4ccc5dfb75b8747fa1a5cd38d8e2d6d24f39fac832c7dbec77e2178635b69c74" Nov 03 16:30:15.997902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:15.997482 5946 scope.go:122] "RemoveContainer" containerID="43096bb3099b9fe2f3d5545afca8c217f5d9b100ee3c08b46bb11c58ff5c6662" Nov 03 16:30:16.998184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:16.998153 5946 scope.go:122] "RemoveContainer" containerID="4d13199bcfdc0fa2952ebb8c7eb1e7c3044dc8c1a2cbe3d1a9bfd2c814c6a99d" Nov 03 16:30:17.313177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.313143 5946 generic.go:356] "Generic (PLEG): container finished" podID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerID="2d3ef35136eeeab43302a6dad596beed7c1eb958777b5fe19c0149577e193457" exitCode=60 Nov 03 16:30:17.313423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.313252 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" event={"ID":"3482258f-d167-4b50-9d1c-6b1c48073fee","Type":"ContainerDied","Data":"2d3ef35136eeeab43302a6dad596beed7c1eb958777b5fe19c0149577e193457"} Nov 03 16:30:17.313423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.313322 5946 scope.go:122] "RemoveContainer" containerID="4ccc5dfb75b8747fa1a5cd38d8e2d6d24f39fac832c7dbec77e2178635b69c74" Nov 03 16:30:17.313892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.313863 5946 scope.go:122] "RemoveContainer" containerID="2d3ef35136eeeab43302a6dad596beed7c1eb958777b5fe19c0149577e193457" Nov 03 16:30:17.314309 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:17.314268 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d_odh-ai-gateway-infra(3482258f-d167-4b50-9d1c-6b1c48073fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" Nov 03 16:30:17.316174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.316149 5946 generic.go:356] "Generic (PLEG): container finished" podID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerID="129a93c59bfcf1ff07e365111ee8eb0eb94c4d19b280d334f27a6122a77e3f88" exitCode=60 Nov 03 16:30:17.316318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.316186 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" event={"ID":"c681a491-07ff-42e3-a4ac-29941b1c4172","Type":"ContainerDied","Data":"129a93c59bfcf1ff07e365111ee8eb0eb94c4d19b280d334f27a6122a77e3f88"} Nov 03 16:30:17.316596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.316574 5946 scope.go:122] "RemoveContainer" containerID="129a93c59bfcf1ff07e365111ee8eb0eb94c4d19b280d334f27a6122a77e3f88" Nov 03 16:30:17.316884 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:17.316841 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847_odh-ai-gateway-infra(c681a491-07ff-42e3-a4ac-29941b1c4172)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" Nov 03 16:30:17.328702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.328683 5946 scope.go:122] "RemoveContainer" containerID="43096bb3099b9fe2f3d5545afca8c217f5d9b100ee3c08b46bb11c58ff5c6662" Nov 03 16:30:17.998197 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.998166 5946 scope.go:122] "RemoveContainer" containerID="30e4feaec0d05bfbf0e514d529e38bc9554210118486bf6a197075763ff58bc7" Nov 03 16:30:17.998665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:17.998252 5946 scope.go:122] "RemoveContainer" containerID="11f8f5dc4edf4a6b84c6fcdac1870e1bceb38e230b78537406eb50f4a99f0d88" Nov 03 16:30:18.330454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.330421 5946 generic.go:356] "Generic (PLEG): container finished" podID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerID="03d9d443c8ebd58ff33d24bfd3a065525dfff474063dabb3887914c24d5fe342" exitCode=60 Nov 03 16:30:18.330585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.330466 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" event={"ID":"caa19ddd-04a8-4e22-b539-8b6b913cbaf8","Type":"ContainerDied","Data":"03d9d443c8ebd58ff33d24bfd3a065525dfff474063dabb3887914c24d5fe342"} Nov 03 16:30:18.330585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.330517 5946 scope.go:122] "RemoveContainer" containerID="4d13199bcfdc0fa2952ebb8c7eb1e7c3044dc8c1a2cbe3d1a9bfd2c814c6a99d" Nov 03 16:30:18.330987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.330965 5946 scope.go:122] "RemoveContainer" containerID="03d9d443c8ebd58ff33d24bfd3a065525dfff474063dabb3887914c24d5fe342" Nov 03 16:30:18.331397 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:18.331370 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk_odh-ai-gateway-infra(caa19ddd-04a8-4e22-b539-8b6b913cbaf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" Nov 03 16:30:18.498757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.498721 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d"] Nov 03 16:30:18.585375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.585168 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847"] Nov 03 16:30:18.663355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.663313 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:18.687998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.687960 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:18.698483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.698448 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3482258f-d167-4b50-9d1c-6b1c48073fee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482258f-d167-4b50-9d1c-6b1c48073fee-maas-api-service-ca\") pod \"3482258f-d167-4b50-9d1c-6b1c48073fee\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " Nov 03 16:30:18.698647 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.698498 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3482258f-d167-4b50-9d1c-6b1c48073fee-kube-api-access-j69f9\" (UniqueName: \"kubernetes.io/projected/3482258f-d167-4b50-9d1c-6b1c48073fee-kube-api-access-j69f9\") pod \"3482258f-d167-4b50-9d1c-6b1c48073fee\" (UID: \"3482258f-d167-4b50-9d1c-6b1c48073fee\") " Nov 03 16:30:18.705257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.700653 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3482258f-d167-4b50-9d1c-6b1c48073fee-maas-api-service-ca" pod "3482258f-d167-4b50-9d1c-6b1c48073fee" (UID: "3482258f-d167-4b50-9d1c-6b1c48073fee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:30:18.717145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.717099 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482258f-d167-4b50-9d1c-6b1c48073fee-kube-api-access-j69f9" pod "3482258f-d167-4b50-9d1c-6b1c48073fee" (UID: "3482258f-d167-4b50-9d1c-6b1c48073fee"). InnerVolumeSpecName "kube-api-access-j69f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:18.750859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.750797 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:18.800388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.800357 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c681a491-07ff-42e3-a4ac-29941b1c4172-kube-api-access-76hjp\" (UniqueName: \"kubernetes.io/projected/c681a491-07ff-42e3-a4ac-29941b1c4172-kube-api-access-76hjp\") pod \"c681a491-07ff-42e3-a4ac-29941b1c4172\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " Nov 03 16:30:18.800576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.800531 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c681a491-07ff-42e3-a4ac-29941b1c4172-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c681a491-07ff-42e3-a4ac-29941b1c4172-maas-api-service-ca\") pod \"c681a491-07ff-42e3-a4ac-29941b1c4172\" (UID: \"c681a491-07ff-42e3-a4ac-29941b1c4172\") " Nov 03 16:30:18.800884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.800853 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c681a491-07ff-42e3-a4ac-29941b1c4172-maas-api-service-ca" pod "c681a491-07ff-42e3-a4ac-29941b1c4172" (UID: "c681a491-07ff-42e3-a4ac-29941b1c4172"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:30:18.801020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.800931 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482258f-d167-4b50-9d1c-6b1c48073fee-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:18.801020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.800948 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69f9\" (UniqueName: \"kubernetes.io/projected/3482258f-d167-4b50-9d1c-6b1c48073fee-kube-api-access-j69f9\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:18.802742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.802718 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c681a491-07ff-42e3-a4ac-29941b1c4172-kube-api-access-76hjp" pod "c681a491-07ff-42e3-a4ac-29941b1c4172" (UID: "c681a491-07ff-42e3-a4ac-29941b1c4172"). InnerVolumeSpecName "kube-api-access-76hjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:18.902944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.902907 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c681a491-07ff-42e3-a4ac-29941b1c4172-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:18.903101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:18.902949 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76hjp\" (UniqueName: \"kubernetes.io/projected/c681a491-07ff-42e3-a4ac-29941b1c4172-kube-api-access-76hjp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:19.338480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.338398 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" Nov 03 16:30:19.338961 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.338401 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847" event={"ID":"c681a491-07ff-42e3-a4ac-29941b1c4172","Type":"ContainerDied","Data":"0b6fd012e3bfb9f52ead028c624740abe02ad2a932e000808eb52763680bfb0e"} Nov 03 16:30:19.338961 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.338538 5946 scope.go:122] "RemoveContainer" containerID="129a93c59bfcf1ff07e365111ee8eb0eb94c4d19b280d334f27a6122a77e3f88" Nov 03 16:30:19.340788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.340765 5946 generic.go:356] "Generic (PLEG): container finished" podID="969c9d64-b1d0-41e7-8589-30600f34c428" containerID="e2903c512876db95321742e4177b4f08c78ad3b6e6b4a23e83d7e65b720e5dca" exitCode=60 Nov 03 16:30:19.340888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.340832 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" event={"ID":"969c9d64-b1d0-41e7-8589-30600f34c428","Type":"ContainerDied","Data":"e2903c512876db95321742e4177b4f08c78ad3b6e6b4a23e83d7e65b720e5dca"} Nov 03 16:30:19.341266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.341243 5946 scope.go:122] "RemoveContainer" containerID="e2903c512876db95321742e4177b4f08c78ad3b6e6b4a23e83d7e65b720e5dca" Nov 03 16:30:19.341606 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:19.341581 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb_odh-ai-gateway-infra(969c9d64-b1d0-41e7-8589-30600f34c428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" Nov 03 16:30:19.342460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.342435 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" Nov 03 16:30:19.342576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.342485 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d" event={"ID":"3482258f-d167-4b50-9d1c-6b1c48073fee","Type":"ContainerDied","Data":"6a550940342211c330703ccb29a7efa39829af4def5e8cfb9a29944e9c115be0"} Nov 03 16:30:19.344781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.344749 5946 generic.go:356] "Generic (PLEG): container finished" podID="04a4104f-4138-4b9d-894f-68c9d2605534" containerID="345df32fc4014bc5d120b1996f122240bfc8c0cb01455143d46f57cca37c9003" exitCode=60 Nov 03 16:30:19.344882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.344820 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" event={"ID":"04a4104f-4138-4b9d-894f-68c9d2605534","Type":"ContainerDied","Data":"345df32fc4014bc5d120b1996f122240bfc8c0cb01455143d46f57cca37c9003"} Nov 03 16:30:19.345239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.345198 5946 scope.go:122] "RemoveContainer" containerID="345df32fc4014bc5d120b1996f122240bfc8c0cb01455143d46f57cca37c9003" Nov 03 16:30:19.345518 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:19.345495 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn_odh-ai-gateway-infra(04a4104f-4138-4b9d-894f-68c9d2605534)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" Nov 03 16:30:19.351642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.351622 5946 scope.go:122] "RemoveContainer" containerID="30e4feaec0d05bfbf0e514d529e38bc9554210118486bf6a197075763ff58bc7" Nov 03 16:30:19.365308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.365290 5946 scope.go:122] "RemoveContainer" containerID="2d3ef35136eeeab43302a6dad596beed7c1eb958777b5fe19c0149577e193457" Nov 03 16:30:19.374815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.374795 5946 scope.go:122] "RemoveContainer" containerID="11f8f5dc4edf4a6b84c6fcdac1870e1bceb38e230b78537406eb50f4a99f0d88" Nov 03 16:30:19.466929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.466885 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk"] Nov 03 16:30:19.622369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.622347 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:19.632507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.632481 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847"] Nov 03 16:30:19.638452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.638418 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895390-t7847"] Nov 03 16:30:19.713623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.713591 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-maas-api-service-ca\") pod \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " Nov 03 16:30:19.713799 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.713699 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-kube-api-access-5vgn6\" (UniqueName: \"kubernetes.io/projected/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-kube-api-access-5vgn6\") pod \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\" (UID: \"caa19ddd-04a8-4e22-b539-8b6b913cbaf8\") " Nov 03 16:30:19.713932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.713910 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-maas-api-service-ca" pod "caa19ddd-04a8-4e22-b539-8b6b913cbaf8" (UID: "caa19ddd-04a8-4e22-b539-8b6b913cbaf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:30:19.714183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.714159 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:19.715946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.715925 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-kube-api-access-5vgn6" pod "caa19ddd-04a8-4e22-b539-8b6b913cbaf8" (UID: "caa19ddd-04a8-4e22-b539-8b6b913cbaf8"). InnerVolumeSpecName "kube-api-access-5vgn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:19.753074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.753040 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d"] Nov 03 16:30:19.761427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.761397 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895390-ph59d"] Nov 03 16:30:19.815662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:19.815634 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vgn6\" (UniqueName: \"kubernetes.io/projected/caa19ddd-04a8-4e22-b539-8b6b913cbaf8-kube-api-access-5vgn6\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:20.009245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.009184 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" path="/var/lib/kubelet/pods/3482258f-d167-4b50-9d1c-6b1c48073fee/volumes" Nov 03 16:30:20.009739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.009716 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" path="/var/lib/kubelet/pods/c681a491-07ff-42e3-a4ac-29941b1c4172/volumes" Nov 03 16:30:20.357908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.357813 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" event={"ID":"caa19ddd-04a8-4e22-b539-8b6b913cbaf8","Type":"ContainerDied","Data":"55c3d2ad53bfc1fba6fdba9e304128ca54d31235957ab505025674ff58f632d2"} Nov 03 16:30:20.357908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.357827 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk" Nov 03 16:30:20.357908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.357874 5946 scope.go:122] "RemoveContainer" containerID="03d9d443c8ebd58ff33d24bfd3a065525dfff474063dabb3887914c24d5fe342" Nov 03 16:30:20.528086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.528057 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb"] Nov 03 16:30:20.533976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.533948 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk"] Nov 03 16:30:20.541569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.541542 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895390-gcxxk"] Nov 03 16:30:20.678685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.678661 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:20.727512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.727485 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969c9d64-b1d0-41e7-8589-30600f34c428-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c9d64-b1d0-41e7-8589-30600f34c428-maas-api-service-ca\") pod \"969c9d64-b1d0-41e7-8589-30600f34c428\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " Nov 03 16:30:20.727685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.727557 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969c9d64-b1d0-41e7-8589-30600f34c428-kube-api-access-l8hz2\" (UniqueName: \"kubernetes.io/projected/969c9d64-b1d0-41e7-8589-30600f34c428-kube-api-access-l8hz2\") pod \"969c9d64-b1d0-41e7-8589-30600f34c428\" (UID: \"969c9d64-b1d0-41e7-8589-30600f34c428\") " Nov 03 16:30:20.727891 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.727862 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969c9d64-b1d0-41e7-8589-30600f34c428-maas-api-service-ca" pod "969c9d64-b1d0-41e7-8589-30600f34c428" (UID: "969c9d64-b1d0-41e7-8589-30600f34c428"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:30:20.727993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.727953 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c9d64-b1d0-41e7-8589-30600f34c428-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:20.729931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.729909 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969c9d64-b1d0-41e7-8589-30600f34c428-kube-api-access-l8hz2" pod "969c9d64-b1d0-41e7-8589-30600f34c428" (UID: "969c9d64-b1d0-41e7-8589-30600f34c428"). InnerVolumeSpecName "kube-api-access-l8hz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:20.829111 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.829072 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8hz2\" (UniqueName: \"kubernetes.io/projected/969c9d64-b1d0-41e7-8589-30600f34c428-kube-api-access-l8hz2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:20.892235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:20.892173 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn"] Nov 03 16:30:21.043967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.043940 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:21.132330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.132296 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a4104f-4138-4b9d-894f-68c9d2605534-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a4104f-4138-4b9d-894f-68c9d2605534-maas-api-service-ca\") pod \"04a4104f-4138-4b9d-894f-68c9d2605534\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " Nov 03 16:30:21.132330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.132338 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a4104f-4138-4b9d-894f-68c9d2605534-kube-api-access-42gzw\" (UniqueName: \"kubernetes.io/projected/04a4104f-4138-4b9d-894f-68c9d2605534-kube-api-access-42gzw\") pod \"04a4104f-4138-4b9d-894f-68c9d2605534\" (UID: \"04a4104f-4138-4b9d-894f-68c9d2605534\") " Nov 03 16:30:21.132678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.132656 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a4104f-4138-4b9d-894f-68c9d2605534-maas-api-service-ca" pod "04a4104f-4138-4b9d-894f-68c9d2605534" (UID: "04a4104f-4138-4b9d-894f-68c9d2605534"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:30:21.132912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.132894 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a4104f-4138-4b9d-894f-68c9d2605534-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:21.134726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.134701 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a4104f-4138-4b9d-894f-68c9d2605534-kube-api-access-42gzw" pod "04a4104f-4138-4b9d-894f-68c9d2605534" (UID: "04a4104f-4138-4b9d-894f-68c9d2605534"). InnerVolumeSpecName "kube-api-access-42gzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:21.233607 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.233581 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gzw\" (UniqueName: \"kubernetes.io/projected/04a4104f-4138-4b9d-894f-68c9d2605534-kube-api-access-42gzw\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:21.368585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.368549 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" event={"ID":"04a4104f-4138-4b9d-894f-68c9d2605534","Type":"ContainerDied","Data":"c03a0b12635d53b3a58d880019d3d6ab305aa57cc215b6d6405b6ca35aad20b9"} Nov 03 16:30:21.368585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.368570 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn" Nov 03 16:30:21.369088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.368615 5946 scope.go:122] "RemoveContainer" containerID="345df32fc4014bc5d120b1996f122240bfc8c0cb01455143d46f57cca37c9003" Nov 03 16:30:21.371503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.371480 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" Nov 03 16:30:21.371612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.371502 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb" event={"ID":"969c9d64-b1d0-41e7-8589-30600f34c428","Type":"ContainerDied","Data":"f7e3905f59b162f577a50e2f683a0a58fd5674e3b3db6309f375ce168351718c"} Nov 03 16:30:21.379189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.379170 5946 scope.go:122] "RemoveContainer" containerID="e2903c512876db95321742e4177b4f08c78ad3b6e6b4a23e83d7e65b720e5dca" Nov 03 16:30:21.420333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.420307 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:21.543780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.543696 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn"] Nov 03 16:30:21.551086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.551058 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895390-g2ltn"] Nov 03 16:30:21.615109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.615076 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:21.665020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.664995 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb"] Nov 03 16:30:21.672126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:21.672099 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895390-ssdfb"] Nov 03 16:30:22.004657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:22.004628 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" path="/var/lib/kubelet/pods/04a4104f-4138-4b9d-894f-68c9d2605534/volumes" Nov 03 16:30:22.005035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:22.005022 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" path="/var/lib/kubelet/pods/969c9d64-b1d0-41e7-8589-30600f34c428/volumes" Nov 03 16:30:22.005407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:22.005395 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" path="/var/lib/kubelet/pods/caa19ddd-04a8-4e22-b539-8b6b913cbaf8/volumes" Nov 03 16:30:22.724306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:22.724261 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:22.918773 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:22.918736 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:23.342632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:23.342591 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:30:23.467973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:23.467938 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:28.269188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:28.269150 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:30.158994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:30.158963 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:30.356080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:30.356046 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:32.970626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:32.970587 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:37.660620 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:37.660583 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:40.449437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:40.449400 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:42.450019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:42.449984 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:42.586406 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:30:42.586368 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:30:44.339594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:44.339561 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:47.272864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:47.272830 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:50.016707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.016676 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z"] Nov 03 16:30:50.017162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017149 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017163 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017174 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017179 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017188 5946 cpu_manager.go:414] "containerMap: removing container" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017192 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017200 5946 cpu_manager.go:414] "containerMap: removing container" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017222 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017228 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017232 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017239 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017244 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017252 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017257 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017263 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017268 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017275 5946 cpu_manager.go:414] "containerMap: removing container" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017279 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017300 5946 cpu_manager.go:414] "containerMap: removing container" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017304 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017310 5946 cpu_manager.go:414] "containerMap: removing container" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017314 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017323 5946 cpu_manager.go:414] "containerMap: removing container" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017327 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017336 5946 cpu_manager.go:414] "containerMap: removing container" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017341 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017428 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017438 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017446 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017460 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017465 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017472 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017478 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017487 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017492 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017498 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017504 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:50.017534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.017509 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3482258f-d167-4b50-9d1c-6b1c48073fee" containerName="cleanup" Nov 03 16:30:50.022683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.022666 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.080323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.080292 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z"] Nov 03 16:30:50.084926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.084900 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.085048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.084936 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzph\" (UniqueName: \"kubernetes.io/projected/9f998d91-333b-43fc-a7a2-79ff8a974c60-kube-api-access-smzph\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.085048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.084957 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.127258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.127224 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:30:50.127258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.127237 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:30:50.127463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.127262 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:30:50.128334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.128305 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:30:50.186841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.186810 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.187028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.186845 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzph\" (UniqueName: \"kubernetes.io/projected/9f998d91-333b-43fc-a7a2-79ff8a974c60-kube-api-access-smzph\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.187028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.186868 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.187550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.187523 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.189548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.189522 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.247895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.247857 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzph\" (UniqueName: \"kubernetes.io/projected/9f998d91-333b-43fc-a7a2-79ff8a974c60-kube-api-access-smzph\") pod \"maas-api-revoke-sub-b7e01dbe-t545z\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.335826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.335753 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:50.718092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:50.718062 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z"] Nov 03 16:30:50.718604 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:30:50.718574 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f998d91_333b_43fc_a7a2_79ff8a974c60.slice/crio-18e6b8c952905dc6b7448f125e019eea48d1811a10caca7451c2e28d7b3118c2 WatchSource:0}: Error finding container 18e6b8c952905dc6b7448f125e019eea48d1811a10caca7451c2e28d7b3118c2: Status 404 returned error can't find the container with id 18e6b8c952905dc6b7448f125e019eea48d1811a10caca7451c2e28d7b3118c2 Nov 03 16:30:51.632053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:51.632017 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" event={"ID":"9f998d91-333b-43fc-a7a2-79ff8a974c60","Type":"ContainerStarted","Data":"dc1a36c1a6b4f0d56a112c5ae0c1d52c7695c624727612c9062d68ff88a272bc"} Nov 03 16:30:51.632455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:51.632106 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" event={"ID":"9f998d91-333b-43fc-a7a2-79ff8a974c60","Type":"ContainerStarted","Data":"18e6b8c952905dc6b7448f125e019eea48d1811a10caca7451c2e28d7b3118c2"} Nov 03 16:30:51.959714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:51.959614 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:52.265245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265183 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x"] Nov 03 16:30:52.265805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265790 5946 cpu_manager.go:414] "containerMap: removing container" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:52.265856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265808 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:52.265856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265848 5946 cpu_manager.go:414] "containerMap: removing container" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:52.265856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265853 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa19ddd-04a8-4e22-b539-8b6b913cbaf8" containerName="cleanup" Nov 03 16:30:52.265977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265965 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a4104f-4138-4b9d-894f-68c9d2605534" containerName="cleanup" Nov 03 16:30:52.266010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265984 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c9d64-b1d0-41e7-8589-30600f34c428" containerName="cleanup" Nov 03 16:30:52.266010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.265997 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c681a491-07ff-42e3-a4ac-29941b1c4172" containerName="cleanup" Nov 03 16:30:52.269542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.269524 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.329767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.329736 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x"] Nov 03 16:30:52.369529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.369500 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"kube-root-ca.crt\"" Nov 03 16:30:52.369689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.369563 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"openshift-service-ca.crt\"" Nov 03 16:30:52.376083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.376047 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"distinct-2-w2-c88d54-kserve-self-signed-certs\"" Nov 03 16:30:52.376275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.376256 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"default-dockercfg-wr497\"" Nov 03 16:30:52.417467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.417427 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:52.420869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.420846 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tls-certs\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.421013 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.420996 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-home\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.421055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.421045 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-model-cache\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.421149 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.421134 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wrm\" (UniqueName: \"kubernetes.io/projected/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-kube-api-access-t7wrm\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.421218 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.421194 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-dshm\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.421316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.421300 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tmp-dir\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.522713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.522631 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tls-certs\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.522888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.522723 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-home\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.522888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.522747 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-model-cache\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.522888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.522831 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wrm\" (UniqueName: \"kubernetes.io/projected/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-kube-api-access-t7wrm\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.522888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.522882 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-dshm\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.523113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.522918 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tmp-dir\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.523285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.523264 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-model-cache\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.523362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.523263 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-home\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.523625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.523558 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tmp-dir\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.525478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.525444 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-dshm\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.525572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.525515 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tls-certs\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.583196 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.583163 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wrm\" (UniqueName: \"kubernetes.io/projected/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-kube-api-access-t7wrm\") pod \"distinct-2-w2-c88d54-kserve-687669d5c8-qns4x\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:52.678253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.678197 5946 generic.go:356] "Generic (PLEG): container finished" podID="9f998d91-333b-43fc-a7a2-79ff8a974c60" containerID="dc1a36c1a6b4f0d56a112c5ae0c1d52c7695c624727612c9062d68ff88a272bc" exitCode=0 Nov 03 16:30:52.679188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.678921 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" event={"ID":"9f998d91-333b-43fc-a7a2-79ff8a974c60","Type":"ContainerDied","Data":"dc1a36c1a6b4f0d56a112c5ae0c1d52c7695c624727612c9062d68ff88a272bc"} Nov 03 16:30:52.873798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.873770 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:52.882573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:52.882551 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:53.033938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.033899 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-xks-ca\") pod \"9f998d91-333b-43fc-a7a2-79ff8a974c60\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " Nov 03 16:30:53.034133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.034111 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-openshift-ca\") pod \"9f998d91-333b-43fc-a7a2-79ff8a974c60\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " Nov 03 16:30:53.034190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.034152 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f998d91-333b-43fc-a7a2-79ff8a974c60-kube-api-access-smzph\" (UniqueName: \"kubernetes.io/projected/9f998d91-333b-43fc-a7a2-79ff8a974c60-kube-api-access-smzph\") pod \"9f998d91-333b-43fc-a7a2-79ff8a974c60\" (UID: \"9f998d91-333b-43fc-a7a2-79ff8a974c60\") " Nov 03 16:30:53.034488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.034465 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-openshift-ca" pod "9f998d91-333b-43fc-a7a2-79ff8a974c60" (UID: "9f998d91-333b-43fc-a7a2-79ff8a974c60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:30:53.034688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.034671 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:53.036044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.036024 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-xks-ca" pod "9f998d91-333b-43fc-a7a2-79ff8a974c60" (UID: "9f998d91-333b-43fc-a7a2-79ff8a974c60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:30:53.036300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.036279 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f998d91-333b-43fc-a7a2-79ff8a974c60-kube-api-access-smzph" pod "9f998d91-333b-43fc-a7a2-79ff8a974c60" (UID: "9f998d91-333b-43fc-a7a2-79ff8a974c60"). InnerVolumeSpecName "kube-api-access-smzph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:53.136151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.136044 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzph\" (UniqueName: \"kubernetes.io/projected/9f998d91-333b-43fc-a7a2-79ff8a974c60-kube-api-access-smzph\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:53.136151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.136080 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f998d91-333b-43fc-a7a2-79ff8a974c60-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:30:53.264166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.264136 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x"] Nov 03 16:30:53.264618 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:30:53.264589 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12dee1e9_3e3f_4bb0_99e0_baf51e1d8e44.slice/crio-0ae35160db49542b6f8d31450de7bfdf34a53d33e90c615a8b387a06db65893d WatchSource:0}: Error finding container 0ae35160db49542b6f8d31450de7bfdf34a53d33e90c615a8b387a06db65893d: Status 404 returned error can't find the container with id 0ae35160db49542b6f8d31450de7bfdf34a53d33e90c615a8b387a06db65893d Nov 03 16:30:53.688733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.688686 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" event={"ID":"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44","Type":"ContainerStarted","Data":"c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb"} Nov 03 16:30:53.689188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.688746 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" event={"ID":"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44","Type":"ContainerStarted","Data":"0ae35160db49542b6f8d31450de7bfdf34a53d33e90c615a8b387a06db65893d"} Nov 03 16:30:53.690177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.690156 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" Nov 03 16:30:53.690349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.690223 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z" event={"ID":"9f998d91-333b-43fc-a7a2-79ff8a974c60","Type":"ContainerDied","Data":"18e6b8c952905dc6b7448f125e019eea48d1811a10caca7451c2e28d7b3118c2"} Nov 03 16:30:53.690349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.690277 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e6b8c952905dc6b7448f125e019eea48d1811a10caca7451c2e28d7b3118c2" Nov 03 16:30:53.809633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.809524 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podStartSLOduration=1.80950954 podStartE2EDuration="1.80950954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:53.808509605 +0000 UTC m=+1123.929562430" watchObservedRunningTime="2026-11-03 16:30:53.80950954 +0000 UTC m=+1123.930562362" Nov 03 16:30:53.883126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:53.883080 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:30:56.769357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:30:56.769320 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:00.693282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.693230 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:00.772992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.772951 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm"] Nov 03 16:31:00.773516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.773499 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9f998d91-333b-43fc-a7a2-79ff8a974c60" containerName="revoke-keys" Nov 03 16:31:00.773516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.773516 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f998d91-333b-43fc-a7a2-79ff8a974c60" containerName="revoke-keys" Nov 03 16:31:00.773674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.773614 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f998d91-333b-43fc-a7a2-79ff8a974c60" containerName="revoke-keys" Nov 03 16:31:00.777258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.777237 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.826986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.826943 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.827161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.827107 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.827242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.827169 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vmv\" (UniqueName: \"kubernetes.io/projected/34627a78-1e05-4fa1-9b89-ea2b386bc999-kube-api-access-p7vmv\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.833944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.833922 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:31:00.839259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.839232 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm"] Nov 03 16:31:00.884640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.884602 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:31:00.884823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.884650 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:31:00.941231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.930981 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.941231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.931076 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vmv\" (UniqueName: \"kubernetes.io/projected/34627a78-1e05-4fa1-9b89-ea2b386bc999-kube-api-access-p7vmv\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.941231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.931231 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.941231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.932065 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:00.946656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.946565 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:31:00.970228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:00.969433 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:01.009962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:01.009917 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vmv\" (UniqueName: \"kubernetes.io/projected/34627a78-1e05-4fa1-9b89-ea2b386bc999-kube-api-access-p7vmv\") pod \"maas-api-revoke-sub-369ef4ba-qzsgm\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:01.091454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:01.091421 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:01.479179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:01.479141 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm"] Nov 03 16:31:01.479489 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:31:01.479459 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34627a78_1e05_4fa1_9b89_ea2b386bc999.slice/crio-08e0fb54f53e87d88fe45d1154483a4a11a2c2bdba439c3f57168a690103d498 WatchSource:0}: Error finding container 08e0fb54f53e87d88fe45d1154483a4a11a2c2bdba439c3f57168a690103d498: Status 404 returned error can't find the container with id 08e0fb54f53e87d88fe45d1154483a4a11a2c2bdba439c3f57168a690103d498 Nov 03 16:31:01.717629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:01.717593 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:01.761556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:01.761517 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" event={"ID":"34627a78-1e05-4fa1-9b89-ea2b386bc999","Type":"ContainerStarted","Data":"08e0fb54f53e87d88fe45d1154483a4a11a2c2bdba439c3f57168a690103d498"} Nov 03 16:31:02.776872 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:02.776828 5946 generic.go:356] "Generic (PLEG): container finished" podID="34627a78-1e05-4fa1-9b89-ea2b386bc999" containerID="67af41cdea1f68edd113b2fbae1500d2331c855514bd951e7c9f78706e513ca3" exitCode=0 Nov 03 16:31:02.777362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:02.777048 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" event={"ID":"34627a78-1e05-4fa1-9b89-ea2b386bc999","Type":"ContainerDied","Data":"67af41cdea1f68edd113b2fbae1500d2331c855514bd951e7c9f78706e513ca3"} Nov 03 16:31:02.883270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:02.883238 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:31:02.897723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:02.897687 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:31:03.798775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.798748 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:31:03.926130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.926097 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:03.975139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.975108 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34627a78-1e05-4fa1-9b89-ea2b386bc999-kube-api-access-p7vmv\" (UniqueName: \"kubernetes.io/projected/34627a78-1e05-4fa1-9b89-ea2b386bc999-kube-api-access-p7vmv\") pod \"34627a78-1e05-4fa1-9b89-ea2b386bc999\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " Nov 03 16:31:03.975337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.975243 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-openshift-ca\") pod \"34627a78-1e05-4fa1-9b89-ea2b386bc999\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " Nov 03 16:31:03.975482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.975464 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-xks-ca\") pod \"34627a78-1e05-4fa1-9b89-ea2b386bc999\" (UID: \"34627a78-1e05-4fa1-9b89-ea2b386bc999\") " Nov 03 16:31:03.975646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.975617 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-openshift-ca" pod "34627a78-1e05-4fa1-9b89-ea2b386bc999" (UID: "34627a78-1e05-4fa1-9b89-ea2b386bc999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:31:03.975930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.975910 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:03.977512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.977487 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34627a78-1e05-4fa1-9b89-ea2b386bc999-kube-api-access-p7vmv" pod "34627a78-1e05-4fa1-9b89-ea2b386bc999" (UID: "34627a78-1e05-4fa1-9b89-ea2b386bc999"). InnerVolumeSpecName "kube-api-access-p7vmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:03.977604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:03.977585 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-xks-ca" pod "34627a78-1e05-4fa1-9b89-ea2b386bc999" (UID: "34627a78-1e05-4fa1-9b89-ea2b386bc999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:04.077582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:04.077497 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34627a78-1e05-4fa1-9b89-ea2b386bc999-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:04.077582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:04.077525 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7vmv\" (UniqueName: \"kubernetes.io/projected/34627a78-1e05-4fa1-9b89-ea2b386bc999-kube-api-access-p7vmv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:04.255786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:04.255750 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:04.793490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:04.793465 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" Nov 03 16:31:04.793672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:04.793463 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm" event={"ID":"34627a78-1e05-4fa1-9b89-ea2b386bc999","Type":"ContainerDied","Data":"08e0fb54f53e87d88fe45d1154483a4a11a2c2bdba439c3f57168a690103d498"} Nov 03 16:31:04.793672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:04.793600 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08e0fb54f53e87d88fe45d1154483a4a11a2c2bdba439c3f57168a690103d498" Nov 03 16:31:05.403940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:05.403906 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:31:06.575235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:06.572842 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:07.087810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:07.087775 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht"] Nov 03 16:31:07.139483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:07.139454 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce6ee70-rhjht"] Nov 03 16:31:08.005238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:08.005187 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b3fd26-7309-492e-a67f-6ad9106473e2" path="/var/lib/kubelet/pods/d1b3fd26-7309-492e-a67f-6ad9106473e2/volumes" Nov 03 16:31:09.054352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:09.054322 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:09.252043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:09.252011 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:11.401517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:11.401474 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:12.586975 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:31:12.586935 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:31:13.084965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:13.084936 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r"] Nov 03 16:31:13.088076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:13.088054 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd3ce74-9dh7r"] Nov 03 16:31:13.111505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:13.111464 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:14.004148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:14.004116 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7b0715-d949-4b9f-afb1-ec7499b9b6d5" path="/var/lib/kubelet/pods/fa7b0715-d949-4b9f-afb1-ec7499b9b6d5/volumes" Nov 03 16:31:16.279923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:16.279889 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:19.061302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.061266 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn"] Nov 03 16:31:19.061815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.061801 5946 cpu_manager.go:414] "containerMap: removing container" podUID="34627a78-1e05-4fa1-9b89-ea2b386bc999" containerName="revoke-keys" Nov 03 16:31:19.061859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.061817 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34627a78-1e05-4fa1-9b89-ea2b386bc999" containerName="revoke-keys" Nov 03 16:31:19.061913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.061904 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="34627a78-1e05-4fa1-9b89-ea2b386bc999" containerName="revoke-keys" Nov 03 16:31:19.067063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.067043 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.089118 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.089094 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv"] Nov 03 16:31:19.092911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.092896 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.128518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.128486 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn"] Nov 03 16:31:19.134465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.134440 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:31:19.134632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.134542 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:31:19.134632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.134588 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:31:19.172954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.172914 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:31:19.173590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.173572 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6ae690\"/\"distinct-2-w5-6ae690-kserve-self-signed-certs\"" Nov 03 16:31:19.174385 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.174363 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.174468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.174455 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz76g\" (UniqueName: \"kubernetes.io/projected/4f216f28-08a0-4051-b711-0c8f92f3ee14-kube-api-access-gz76g\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.174631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.174604 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.177763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.177744 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv"] Nov 03 16:31:19.214996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.214964 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:19.276314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276276 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.276512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276331 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-home\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.276512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276423 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-model-cache\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.276599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276516 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz76g\" (UniqueName: \"kubernetes.io/projected/4f216f28-08a0-4051-b711-0c8f92f3ee14-kube-api-access-gz76g\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.276599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276576 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424b9fdd-1579-4162-827c-e48d0c5af5c9-tls-certs\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.276685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276609 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.276743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276673 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-tmp-dir\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.276803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276744 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-dshm\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.276803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.276793 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm766\" (UniqueName: \"kubernetes.io/projected/424b9fdd-1579-4162-827c-e48d0c5af5c9-kube-api-access-vm766\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.277138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.277113 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.279578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.279552 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.339301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.339220 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz76g\" (UniqueName: \"kubernetes.io/projected/4f216f28-08a0-4051-b711-0c8f92f3ee14-kube-api-access-gz76g\") pod \"maas-api-revoke-sub-738b2dd0-cl8mn\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.377631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.377588 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-model-cache\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.377812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.377652 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424b9fdd-1579-4162-827c-e48d0c5af5c9-tls-certs\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.377812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.377675 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-tmp-dir\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.377812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.377707 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-dshm\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.377812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.377744 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm766\" (UniqueName: \"kubernetes.io/projected/424b9fdd-1579-4162-827c-e48d0c5af5c9-kube-api-access-vm766\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.378060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.377848 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-home\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.378060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.378047 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-model-cache\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.378336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.378311 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-home\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.378435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.378338 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-tmp-dir\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.380343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.380316 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-dshm\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.380484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.380464 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424b9fdd-1579-4162-827c-e48d0c5af5c9-tls-certs\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.382450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.382427 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:19.440916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.440886 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm766\" (UniqueName: \"kubernetes.io/projected/424b9fdd-1579-4162-827c-e48d0c5af5c9-kube-api-access-vm766\") pod \"distinct-2-w5-6ae690-kserve-655676bfd4-2spvv\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.704998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.704907 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:19.777844 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:31:19.777798 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f216f28_08a0_4051_b711_0c8f92f3ee14.slice/crio-c48f90af3a337bf1a1a83e265dfdf8ef9e71c48d2508488f66d57af706225f0a WatchSource:0}: Error finding container c48f90af3a337bf1a1a83e265dfdf8ef9e71c48d2508488f66d57af706225f0a: Status 404 returned error can't find the container with id c48f90af3a337bf1a1a83e265dfdf8ef9e71c48d2508488f66d57af706225f0a Nov 03 16:31:19.780159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.780131 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn"] Nov 03 16:31:19.928508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:19.928473 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" event={"ID":"4f216f28-08a0-4051-b711-0c8f92f3ee14","Type":"ContainerStarted","Data":"c48f90af3a337bf1a1a83e265dfdf8ef9e71c48d2508488f66d57af706225f0a"} Nov 03 16:31:20.096057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:20.096031 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv"] Nov 03 16:31:20.097771 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:31:20.097747 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod424b9fdd_1579_4162_827c_e48d0c5af5c9.slice/crio-19ce76f98d921a488440e7a3efca3d1ade480bc1bdec38b1feb44b9aef01c309 WatchSource:0}: Error finding container 19ce76f98d921a488440e7a3efca3d1ade480bc1bdec38b1feb44b9aef01c309: Status 404 returned error can't find the container with id 19ce76f98d921a488440e7a3efca3d1ade480bc1bdec38b1feb44b9aef01c309 Nov 03 16:31:20.938010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:20.937915 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" event={"ID":"424b9fdd-1579-4162-827c-e48d0c5af5c9","Type":"ContainerStarted","Data":"32d280cb2c7e356d1e9b25a1ff2678189dda4ac510d38d1d2eb6d03d4f6b2752"} Nov 03 16:31:20.938010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:20.937960 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" event={"ID":"424b9fdd-1579-4162-827c-e48d0c5af5c9","Type":"ContainerStarted","Data":"19ce76f98d921a488440e7a3efca3d1ade480bc1bdec38b1feb44b9aef01c309"} Nov 03 16:31:20.939623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:20.939589 5946 generic.go:356] "Generic (PLEG): container finished" podID="4f216f28-08a0-4051-b711-0c8f92f3ee14" containerID="3b0c20887077d13e967aaa9c7d8578605a9d68f05faccc623ebde1a6dccc0ca1" exitCode=0 Nov 03 16:31:20.939732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:20.939679 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" event={"ID":"4f216f28-08a0-4051-b711-0c8f92f3ee14","Type":"ContainerDied","Data":"3b0c20887077d13e967aaa9c7d8578605a9d68f05faccc623ebde1a6dccc0ca1"} Nov 03 16:31:21.066766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:21.066704 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:21.201547 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:21.201395 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podStartSLOduration=2.201375803 podStartE2EDuration="2.201375803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:21.065998032 +0000 UTC m=+1151.187050858" watchObservedRunningTime="2026-11-03 16:31:21.201375803 +0000 UTC m=+1151.322428627" Nov 03 16:31:21.705668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:21.705640 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:22.102374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.102347 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:22.212943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.212911 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-openshift-ca\") pod \"4f216f28-08a0-4051-b711-0c8f92f3ee14\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " Nov 03 16:31:22.213444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.213017 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f216f28-08a0-4051-b711-0c8f92f3ee14-kube-api-access-gz76g\" (UniqueName: \"kubernetes.io/projected/4f216f28-08a0-4051-b711-0c8f92f3ee14-kube-api-access-gz76g\") pod \"4f216f28-08a0-4051-b711-0c8f92f3ee14\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " Nov 03 16:31:22.213444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.213291 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-openshift-ca" pod "4f216f28-08a0-4051-b711-0c8f92f3ee14" (UID: "4f216f28-08a0-4051-b711-0c8f92f3ee14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:31:22.213444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.213301 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-xks-ca\") pod \"4f216f28-08a0-4051-b711-0c8f92f3ee14\" (UID: \"4f216f28-08a0-4051-b711-0c8f92f3ee14\") " Nov 03 16:31:22.213756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.213732 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:22.215344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.215319 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-xks-ca" pod "4f216f28-08a0-4051-b711-0c8f92f3ee14" (UID: "4f216f28-08a0-4051-b711-0c8f92f3ee14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:22.215431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.215341 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f216f28-08a0-4051-b711-0c8f92f3ee14-kube-api-access-gz76g" pod "4f216f28-08a0-4051-b711-0c8f92f3ee14" (UID: "4f216f28-08a0-4051-b711-0c8f92f3ee14"). InnerVolumeSpecName "kube-api-access-gz76g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:22.315275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.315183 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f216f28-08a0-4051-b711-0c8f92f3ee14-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:22.315275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.315237 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz76g\" (UniqueName: \"kubernetes.io/projected/4f216f28-08a0-4051-b711-0c8f92f3ee14-kube-api-access-gz76g\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:22.958800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.958759 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" Nov 03 16:31:22.958800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.958787 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn" event={"ID":"4f216f28-08a0-4051-b711-0c8f92f3ee14","Type":"ContainerDied","Data":"c48f90af3a337bf1a1a83e265dfdf8ef9e71c48d2508488f66d57af706225f0a"} Nov 03 16:31:22.958994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:22.958839 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48f90af3a337bf1a1a83e265dfdf8ef9e71c48d2508488f66d57af706225f0a" Nov 03 16:31:25.849563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:25.849528 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:27.063315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:27.063285 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2"] Nov 03 16:31:27.099937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:27.099909 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48df6cf2-pqhb2"] Nov 03 16:31:28.004522 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:28.004486 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c" path="/var/lib/kubelet/pods/aa18c3f5-a2f1-4f13-97c7-c5a79e14b38c/volumes" Nov 03 16:31:28.078216 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:28.078167 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4"] Nov 03 16:31:28.129251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:28.129217 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c436991-5djd4"] Nov 03 16:31:29.705159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:29.705125 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:29.718076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:29.718049 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:30.005569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.005533 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5434235a-5555-4b2a-ac85-87c33495e2b4" path="/var/lib/kubelet/pods/5434235a-5555-4b2a-ac85-87c33495e2b4/volumes" Nov 03 16:31:30.030134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.030108 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:31:30.499241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.499180 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:30.691221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.691166 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:30.861817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.861722 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t"] Nov 03 16:31:30.862392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.862375 5946 cpu_manager.go:414] "containerMap: removing container" podUID="4f216f28-08a0-4051-b711-0c8f92f3ee14" containerName="revoke-keys" Nov 03 16:31:30.862449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.862393 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f216f28-08a0-4051-b711-0c8f92f3ee14" containerName="revoke-keys" Nov 03 16:31:30.862504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.862495 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f216f28-08a0-4051-b711-0c8f92f3ee14" containerName="revoke-keys" Nov 03 16:31:30.867191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.867174 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917363 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917403 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917531 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917604 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zsvq\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-kube-api-access-2zsvq\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917679 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917721 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917848 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917879 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.917980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.917964 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:30.922942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.922917 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bcd756-openshift-default-dockercfg-8drrh\"" Nov 03 16:31:30.976653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:30.976616 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t"] Nov 03 16:31:31.018838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.018804 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zsvq\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-kube-api-access-2zsvq\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.018855 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.018895 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.018987 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019015 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019054 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019125 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019151 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019266 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019419 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019432 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019638 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019768 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.019841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.019786 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.021826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.021801 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.022078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.022059 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.078873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.078845 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.079089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.079054 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zsvq\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-kube-api-access-2zsvq\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.181098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.181011 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:31.617844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.617818 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t"] Nov 03 16:31:31.618692 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:31:31.618667 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6fe0ae_63e2_4c93_9aca_f532d34e9b10.slice/crio-d47c21087665e95c63f9bd9a592ff432cbe8bc211a7de1425d23ffaedd13c9c3 WatchSource:0}: Error finding container d47c21087665e95c63f9bd9a592ff432cbe8bc211a7de1425d23ffaedd13c9c3: Status 404 returned error can't find the container with id d47c21087665e95c63f9bd9a592ff432cbe8bc211a7de1425d23ffaedd13c9c3 Nov 03 16:31:31.621095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.621064 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:31:31.621151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.621131 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:31:31.621191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:31.621159 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:31:32.036081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:32.036041 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" event={"ID":"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10","Type":"ContainerStarted","Data":"c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74"} Nov 03 16:31:32.036484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:32.036102 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" event={"ID":"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10","Type":"ContainerStarted","Data":"d47c21087665e95c63f9bd9a592ff432cbe8bc211a7de1425d23ffaedd13c9c3"} Nov 03 16:31:32.159543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:32.159486 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podStartSLOduration=2.159471744 podStartE2EDuration="2.159471744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:32.158169163 +0000 UTC m=+1162.279221998" watchObservedRunningTime="2026-11-03 16:31:32.159471744 +0000 UTC m=+1162.280524566" Nov 03 16:31:32.181883 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:32.181851 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:32.183672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:32.183648 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:31:32.183751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:32.183695 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 16:31:33.182023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.181989 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:31:33.182450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.182041 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 16:31:33.845261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.845223 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w"] Nov 03 16:31:33.850761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.850734 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:33.863402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.863369 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x78q\" (UniqueName: \"kubernetes.io/projected/76923b62-022f-4ab6-a3bd-66b431630e66-kube-api-access-8x78q\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:33.863752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.863729 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:33.863899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.863882 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:33.911869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.911835 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w"] Nov 03 16:31:33.961168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.961138 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:31:33.961480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.961464 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:31:33.961809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.961793 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:31:33.962017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.962004 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:31:33.991902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.991871 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:33.992201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:33.992177 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:34.008421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.007567 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:34.009246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.008729 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x78q\" (UniqueName: \"kubernetes.io/projected/76923b62-022f-4ab6-a3bd-66b431630e66-kube-api-access-8x78q\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:34.069839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.045751 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:34.104198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.099977 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x78q\" (UniqueName: \"kubernetes.io/projected/76923b62-022f-4ab6-a3bd-66b431630e66-kube-api-access-8x78q\") pod \"maas-api-revoke-sub-b7d968fd-rz49w\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:34.184646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.184608 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:31:34.185231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.185175 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 16:31:34.212201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.212165 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:34.813234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:34.808634 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w"] Nov 03 16:31:35.176913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:35.176811 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" event={"ID":"76923b62-022f-4ab6-a3bd-66b431630e66","Type":"ContainerStarted","Data":"5f273fcd572eee2ec08218389c120f5de8feccde6211cc6e3be2823d31389596"} Nov 03 16:31:35.182715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:35.182679 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:31:35.182971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:35.182942 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 16:31:36.184849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:36.184802 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:31:36.185535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:36.185477 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 16:31:36.223169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:36.221758 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" event={"ID":"76923b62-022f-4ab6-a3bd-66b431630e66","Type":"ContainerStarted","Data":"44b922d79015544449a9573ff78e66947ab3b876e6514fc4d4480ccd07e2dfa9"} Nov 03 16:31:36.226809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:36.223878 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:36.352081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:36.352015 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" podStartSLOduration=3.352000003 podStartE2EDuration="3.352000003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:36.349564588 +0000 UTC m=+1166.470617411" watchObservedRunningTime="2026-11-03 16:31:36.352000003 +0000 UTC m=+1166.473052825" Nov 03 16:31:37.181675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:37.181633 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:31:37.181836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:37.181701 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 16:31:37.248779 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:37.248737 5946 generic.go:356] "Generic (PLEG): container finished" podID="76923b62-022f-4ab6-a3bd-66b431630e66" containerID="44b922d79015544449a9573ff78e66947ab3b876e6514fc4d4480ccd07e2dfa9" exitCode=0 Nov 03 16:31:37.249270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:37.248832 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" event={"ID":"76923b62-022f-4ab6-a3bd-66b431630e66","Type":"ContainerDied","Data":"44b922d79015544449a9573ff78e66947ab3b876e6514fc4d4480ccd07e2dfa9"} Nov 03 16:31:38.181637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.181598 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:31:38.181809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.181672 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 16:31:38.443936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.443873 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:38.562805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.562774 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-xks-ca\") pod \"76923b62-022f-4ab6-a3bd-66b431630e66\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " Nov 03 16:31:38.562999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.562886 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76923b62-022f-4ab6-a3bd-66b431630e66-kube-api-access-8x78q\" (UniqueName: \"kubernetes.io/projected/76923b62-022f-4ab6-a3bd-66b431630e66-kube-api-access-8x78q\") pod \"76923b62-022f-4ab6-a3bd-66b431630e66\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " Nov 03 16:31:38.562999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.562965 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-openshift-ca\") pod \"76923b62-022f-4ab6-a3bd-66b431630e66\" (UID: \"76923b62-022f-4ab6-a3bd-66b431630e66\") " Nov 03 16:31:38.563379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.563353 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-openshift-ca" pod "76923b62-022f-4ab6-a3bd-66b431630e66" (UID: "76923b62-022f-4ab6-a3bd-66b431630e66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:31:38.563547 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.563528 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:38.565299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.565260 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76923b62-022f-4ab6-a3bd-66b431630e66-kube-api-access-8x78q" pod "76923b62-022f-4ab6-a3bd-66b431630e66" (UID: "76923b62-022f-4ab6-a3bd-66b431630e66"). InnerVolumeSpecName "kube-api-access-8x78q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:38.566317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.566296 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-xks-ca" pod "76923b62-022f-4ab6-a3bd-66b431630e66" (UID: "76923b62-022f-4ab6-a3bd-66b431630e66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:38.664340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.664311 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76923b62-022f-4ab6-a3bd-66b431630e66-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:38.664340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:38.664338 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x78q\" (UniqueName: \"kubernetes.io/projected/76923b62-022f-4ab6-a3bd-66b431630e66-kube-api-access-8x78q\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:39.268002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:39.267970 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" Nov 03 16:31:39.268002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:39.267989 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w" event={"ID":"76923b62-022f-4ab6-a3bd-66b431630e66","Type":"ContainerDied","Data":"5f273fcd572eee2ec08218389c120f5de8feccde6211cc6e3be2823d31389596"} Nov 03 16:31:39.268249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:39.268040 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f273fcd572eee2ec08218389c120f5de8feccde6211cc6e3be2823d31389596" Nov 03 16:31:39.380345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:39.380299 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:39.380972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:39.380773 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:39.382092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:39.382070 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:31:41.184341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:41.184286 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:42.586003 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:31:42.585968 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:31:45.865644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:45.865594 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:48.504387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.504353 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs"] Nov 03 16:31:48.505119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.505099 5946 cpu_manager.go:414] "containerMap: removing container" podUID="76923b62-022f-4ab6-a3bd-66b431630e66" containerName="revoke-keys" Nov 03 16:31:48.505266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.505121 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76923b62-022f-4ab6-a3bd-66b431630e66" containerName="revoke-keys" Nov 03 16:31:48.505341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.505329 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="76923b62-022f-4ab6-a3bd-66b431630e66" containerName="revoke-keys" Nov 03 16:31:48.508879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.508857 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.569448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.569413 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs"] Nov 03 16:31:48.589077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.589042 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.589285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.589248 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.589383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.589364 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xpg\" (UniqueName: \"kubernetes.io/projected/3bc82628-fec8-48be-8d95-8419851d5e31-kube-api-access-h2xpg\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.614437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.614393 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:31:48.614437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.614402 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:31:48.614437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.614402 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:31:48.614666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.614454 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:31:48.690691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.690659 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.690860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.690750 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xpg\" (UniqueName: \"kubernetes.io/projected/3bc82628-fec8-48be-8d95-8419851d5e31-kube-api-access-h2xpg\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.690860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.690797 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.691507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.691442 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.693181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.693158 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.753807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.753775 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xpg\" (UniqueName: \"kubernetes.io/projected/3bc82628-fec8-48be-8d95-8419851d5e31-kube-api-access-h2xpg\") pod \"maas-api-revoke-sub-0f690c3c-n6dqs\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:48.822951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:48.822873 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:49.074476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:49.074376 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r"] Nov 03 16:31:49.079439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:49.079409 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4b021b-26p2r"] Nov 03 16:31:49.242731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:49.242697 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs"] Nov 03 16:31:49.245818 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:31:49.245791 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc82628_fec8_48be_8d95_8419851d5e31.slice/crio-f1c8d9d1ca290fb21822e77444542c2b1652c450e976380f45481d475e970b4c WatchSource:0}: Error finding container f1c8d9d1ca290fb21822e77444542c2b1652c450e976380f45481d475e970b4c: Status 404 returned error can't find the container with id f1c8d9d1ca290fb21822e77444542c2b1652c450e976380f45481d475e970b4c Nov 03 16:31:49.341293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:49.341196 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" event={"ID":"3bc82628-fec8-48be-8d95-8419851d5e31","Type":"ContainerStarted","Data":"f1c8d9d1ca290fb21822e77444542c2b1652c450e976380f45481d475e970b4c"} Nov 03 16:31:50.004855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.004822 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3fcbb1-7a72-4bba-a064-423790337ce0" path="/var/lib/kubelet/pods/ef3fcbb1-7a72-4bba-a064-423790337ce0/volumes" Nov 03 16:31:50.063897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.063823 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx"] Nov 03 16:31:50.067586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.067556 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18831b-wxqzx"] Nov 03 16:31:50.350131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.350042 5946 generic.go:356] "Generic (PLEG): container finished" podID="3bc82628-fec8-48be-8d95-8419851d5e31" containerID="d0a7cdd6e51957d3c1a265ad23101bddce9b779547e9269a87e0b570e9784541" exitCode=0 Nov 03 16:31:50.350306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.350130 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" event={"ID":"3bc82628-fec8-48be-8d95-8419851d5e31","Type":"ContainerDied","Data":"d0a7cdd6e51957d3c1a265ad23101bddce9b779547e9269a87e0b570e9784541"} Nov 03 16:31:50.380459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.380428 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292"] Nov 03 16:31:50.384501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.384478 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.414161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.414139 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.414306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.414286 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.414376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.414364 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgpp\" (UniqueName: \"kubernetes.io/projected/abe1369b-d064-40fc-9dd1-47519c1a9807-kube-api-access-nvgpp\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.446321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.446295 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292"] Nov 03 16:31:50.515101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.515067 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgpp\" (UniqueName: \"kubernetes.io/projected/abe1369b-d064-40fc-9dd1-47519c1a9807-kube-api-access-nvgpp\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.515315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.515124 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.515315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.515233 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.515794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.515773 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.517744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.517722 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.575805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.575772 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgpp\" (UniqueName: \"kubernetes.io/projected/abe1369b-d064-40fc-9dd1-47519c1a9807-kube-api-access-nvgpp\") pod \"maas-api-revoke-sub-d7b01fde-qq292\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.697936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.697848 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:50.808803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:50.807832 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:51.064706 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.064671 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw"] Nov 03 16:31:51.069871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.069843 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a7c4d6-4j2bw"] Nov 03 16:31:51.094219 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.094171 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292"] Nov 03 16:31:51.128238 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:31:51.128188 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe1369b_d064_40fc_9dd1_47519c1a9807.slice/crio-acd25033fa3ac844896a3ff7645bc99f0d816a4a7fa0b6267ed227f8311fbb3c WatchSource:0}: Error finding container acd25033fa3ac844896a3ff7645bc99f0d816a4a7fa0b6267ed227f8311fbb3c: Status 404 returned error can't find the container with id acd25033fa3ac844896a3ff7645bc99f0d816a4a7fa0b6267ed227f8311fbb3c Nov 03 16:31:51.359739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.359706 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" event={"ID":"abe1369b-d064-40fc-9dd1-47519c1a9807","Type":"ContainerStarted","Data":"acd25033fa3ac844896a3ff7645bc99f0d816a4a7fa0b6267ed227f8311fbb3c"} Nov 03 16:31:51.513227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.513187 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:51.631614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.631528 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-xks-ca\") pod \"3bc82628-fec8-48be-8d95-8419851d5e31\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " Nov 03 16:31:51.631766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.631694 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc82628-fec8-48be-8d95-8419851d5e31-kube-api-access-h2xpg\" (UniqueName: \"kubernetes.io/projected/3bc82628-fec8-48be-8d95-8419851d5e31-kube-api-access-h2xpg\") pod \"3bc82628-fec8-48be-8d95-8419851d5e31\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " Nov 03 16:31:51.631766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.631723 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-openshift-ca\") pod \"3bc82628-fec8-48be-8d95-8419851d5e31\" (UID: \"3bc82628-fec8-48be-8d95-8419851d5e31\") " Nov 03 16:31:51.632149 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.632120 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-openshift-ca" pod "3bc82628-fec8-48be-8d95-8419851d5e31" (UID: "3bc82628-fec8-48be-8d95-8419851d5e31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:31:51.632329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.632227 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:51.633795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.633768 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-xks-ca" pod "3bc82628-fec8-48be-8d95-8419851d5e31" (UID: "3bc82628-fec8-48be-8d95-8419851d5e31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:51.633946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.633923 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc82628-fec8-48be-8d95-8419851d5e31-kube-api-access-h2xpg" pod "3bc82628-fec8-48be-8d95-8419851d5e31" (UID: "3bc82628-fec8-48be-8d95-8419851d5e31"). InnerVolumeSpecName "kube-api-access-h2xpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:51.732951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.732908 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82628-fec8-48be-8d95-8419851d5e31-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:51.732951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:51.732940 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xpg\" (UniqueName: \"kubernetes.io/projected/3bc82628-fec8-48be-8d95-8419851d5e31-kube-api-access-h2xpg\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:52.004689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.004656 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ad70a9-6ab8-4eff-a50b-8dc10155b33a" path="/var/lib/kubelet/pods/22ad70a9-6ab8-4eff-a50b-8dc10155b33a/volumes" Nov 03 16:31:52.005083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.005069 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db7b15ab-7a30-45df-bb25-6f8ed973b0a0" path="/var/lib/kubelet/pods/db7b15ab-7a30-45df-bb25-6f8ed973b0a0/volumes" Nov 03 16:31:52.194115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.194079 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw"] Nov 03 16:31:52.194703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.194686 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3bc82628-fec8-48be-8d95-8419851d5e31" containerName="revoke-keys" Nov 03 16:31:52.194746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.194705 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc82628-fec8-48be-8d95-8419851d5e31" containerName="revoke-keys" Nov 03 16:31:52.194822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.194812 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc82628-fec8-48be-8d95-8419851d5e31" containerName="revoke-keys" Nov 03 16:31:52.198216 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.198186 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.239046 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.239015 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.239253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.239145 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4qq\" (UniqueName: \"kubernetes.io/projected/9bed10d1-6b75-44fe-adaa-0c257a008615-kube-api-access-fl4qq\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.239410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.239393 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.268462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.268376 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw"] Nov 03 16:31:52.340786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.340744 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.341005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.340835 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.341005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.340943 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4qq\" (UniqueName: \"kubernetes.io/projected/9bed10d1-6b75-44fe-adaa-0c257a008615-kube-api-access-fl4qq\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.341577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.341556 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.343573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.343544 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.368410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.368377 5946 generic.go:356] "Generic (PLEG): container finished" podID="abe1369b-d064-40fc-9dd1-47519c1a9807" containerID="6c07fb5a361659bcd39ad7df278b102d9827b196583533e3c8b0f26c164f6758" exitCode=0 Nov 03 16:31:52.368563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.368456 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" event={"ID":"abe1369b-d064-40fc-9dd1-47519c1a9807","Type":"ContainerDied","Data":"6c07fb5a361659bcd39ad7df278b102d9827b196583533e3c8b0f26c164f6758"} Nov 03 16:31:52.369942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.369919 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" event={"ID":"3bc82628-fec8-48be-8d95-8419851d5e31","Type":"ContainerDied","Data":"f1c8d9d1ca290fb21822e77444542c2b1652c450e976380f45481d475e970b4c"} Nov 03 16:31:52.369942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.369936 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs" Nov 03 16:31:52.370100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.369956 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c8d9d1ca290fb21822e77444542c2b1652c450e976380f45481d475e970b4c" Nov 03 16:31:52.401320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.401285 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4qq\" (UniqueName: \"kubernetes.io/projected/9bed10d1-6b75-44fe-adaa-0c257a008615-kube-api-access-fl4qq\") pod \"maas-api-revoke-sub-7fec2afb-phvgw\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.512330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.512303 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:52.899329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:52.899298 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw"] Nov 03 16:31:52.899605 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:31:52.899583 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bed10d1_6b75_44fe_adaa_0c257a008615.slice/crio-ce87c26a8bd509fe4115f74e62e6cca79f6cb0da583458d6357f0565df5aa92e WatchSource:0}: Error finding container ce87c26a8bd509fe4115f74e62e6cca79f6cb0da583458d6357f0565df5aa92e: Status 404 returned error can't find the container with id ce87c26a8bd509fe4115f74e62e6cca79f6cb0da583458d6357f0565df5aa92e Nov 03 16:31:53.379261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.379217 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" event={"ID":"9bed10d1-6b75-44fe-adaa-0c257a008615","Type":"ContainerStarted","Data":"6c50b8c89f2c34e6e825a193b77fcbf3514fb7fedd314f51aad19cf610e965ef"} Nov 03 16:31:53.379661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.379283 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" event={"ID":"9bed10d1-6b75-44fe-adaa-0c257a008615","Type":"ContainerStarted","Data":"ce87c26a8bd509fe4115f74e62e6cca79f6cb0da583458d6357f0565df5aa92e"} Nov 03 16:31:53.534416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.534383 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:53.656975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.656941 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe1369b-d064-40fc-9dd1-47519c1a9807-kube-api-access-nvgpp\" (UniqueName: \"kubernetes.io/projected/abe1369b-d064-40fc-9dd1-47519c1a9807-kube-api-access-nvgpp\") pod \"abe1369b-d064-40fc-9dd1-47519c1a9807\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " Nov 03 16:31:53.657201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.657183 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-xks-ca\") pod \"abe1369b-d064-40fc-9dd1-47519c1a9807\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " Nov 03 16:31:53.657299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.657237 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-openshift-ca\") pod \"abe1369b-d064-40fc-9dd1-47519c1a9807\" (UID: \"abe1369b-d064-40fc-9dd1-47519c1a9807\") " Nov 03 16:31:53.657566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.657534 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-openshift-ca" pod "abe1369b-d064-40fc-9dd1-47519c1a9807" (UID: "abe1369b-d064-40fc-9dd1-47519c1a9807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:31:53.657791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.657772 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:53.659168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.659148 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe1369b-d064-40fc-9dd1-47519c1a9807-kube-api-access-nvgpp" pod "abe1369b-d064-40fc-9dd1-47519c1a9807" (UID: "abe1369b-d064-40fc-9dd1-47519c1a9807"). InnerVolumeSpecName "kube-api-access-nvgpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:53.659320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.659298 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-xks-ca" pod "abe1369b-d064-40fc-9dd1-47519c1a9807" (UID: "abe1369b-d064-40fc-9dd1-47519c1a9807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:53.672911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.672851 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" podStartSLOduration=1.67283644 podStartE2EDuration="1.67283644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:53.551044567 +0000 UTC m=+1183.672097389" watchObservedRunningTime="2026-11-03 16:31:53.67283644 +0000 UTC m=+1183.793889328" Nov 03 16:31:53.758856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.758773 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe1369b-d064-40fc-9dd1-47519c1a9807-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:53.758856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:53.758806 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvgpp\" (UniqueName: \"kubernetes.io/projected/abe1369b-d064-40fc-9dd1-47519c1a9807-kube-api-access-nvgpp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:54.390146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:54.390062 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" Nov 03 16:31:54.390146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:54.390092 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292" event={"ID":"abe1369b-d064-40fc-9dd1-47519c1a9807","Type":"ContainerDied","Data":"acd25033fa3ac844896a3ff7645bc99f0d816a4a7fa0b6267ed227f8311fbb3c"} Nov 03 16:31:54.390146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:54.390141 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd25033fa3ac844896a3ff7645bc99f0d816a4a7fa0b6267ed227f8311fbb3c" Nov 03 16:31:54.392057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:54.392033 5946 generic.go:356] "Generic (PLEG): container finished" podID="9bed10d1-6b75-44fe-adaa-0c257a008615" containerID="6c50b8c89f2c34e6e825a193b77fcbf3514fb7fedd314f51aad19cf610e965ef" exitCode=0 Nov 03 16:31:54.392356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:54.392171 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" event={"ID":"9bed10d1-6b75-44fe-adaa-0c257a008615","Type":"ContainerDied","Data":"6c50b8c89f2c34e6e825a193b77fcbf3514fb7fedd314f51aad19cf610e965ef"} Nov 03 16:31:55.541737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.541715 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:55.584562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.584413 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:55.682215 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.682114 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-openshift-ca\") pod \"9bed10d1-6b75-44fe-adaa-0c257a008615\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " Nov 03 16:31:55.682386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.682230 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-xks-ca\") pod \"9bed10d1-6b75-44fe-adaa-0c257a008615\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " Nov 03 16:31:55.682483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.682454 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-openshift-ca" pod "9bed10d1-6b75-44fe-adaa-0c257a008615" (UID: "9bed10d1-6b75-44fe-adaa-0c257a008615"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:31:55.682544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.682504 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bed10d1-6b75-44fe-adaa-0c257a008615-kube-api-access-fl4qq\" (UniqueName: \"kubernetes.io/projected/9bed10d1-6b75-44fe-adaa-0c257a008615-kube-api-access-fl4qq\") pod \"9bed10d1-6b75-44fe-adaa-0c257a008615\" (UID: \"9bed10d1-6b75-44fe-adaa-0c257a008615\") " Nov 03 16:31:55.683049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.682969 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:55.684463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.684438 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-xks-ca" pod "9bed10d1-6b75-44fe-adaa-0c257a008615" (UID: "9bed10d1-6b75-44fe-adaa-0c257a008615"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:55.684607 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.684583 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bed10d1-6b75-44fe-adaa-0c257a008615-kube-api-access-fl4qq" pod "9bed10d1-6b75-44fe-adaa-0c257a008615" (UID: "9bed10d1-6b75-44fe-adaa-0c257a008615"). InnerVolumeSpecName "kube-api-access-fl4qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:55.784161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.784122 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4qq\" (UniqueName: \"kubernetes.io/projected/9bed10d1-6b75-44fe-adaa-0c257a008615-kube-api-access-fl4qq\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:55.784161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:55.784164 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bed10d1-6b75-44fe-adaa-0c257a008615-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:31:56.409818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:56.409730 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" Nov 03 16:31:56.409818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:56.409781 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw" event={"ID":"9bed10d1-6b75-44fe-adaa-0c257a008615","Type":"ContainerDied","Data":"ce87c26a8bd509fe4115f74e62e6cca79f6cb0da583458d6357f0565df5aa92e"} Nov 03 16:31:56.410044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:56.409850 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce87c26a8bd509fe4115f74e62e6cca79f6cb0da583458d6357f0565df5aa92e" Nov 03 16:31:59.072712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.072682 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg"] Nov 03 16:31:59.123318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.123285 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23261a6-4v6kg"] Nov 03 16:31:59.764846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.764815 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl"] Nov 03 16:31:59.765357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.765342 5946 cpu_manager.go:414] "containerMap: removing container" podUID="abe1369b-d064-40fc-9dd1-47519c1a9807" containerName="revoke-keys" Nov 03 16:31:59.765417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.765358 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe1369b-d064-40fc-9dd1-47519c1a9807" containerName="revoke-keys" Nov 03 16:31:59.765417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.765393 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9bed10d1-6b75-44fe-adaa-0c257a008615" containerName="revoke-keys" Nov 03 16:31:59.765417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.765399 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bed10d1-6b75-44fe-adaa-0c257a008615" containerName="revoke-keys" Nov 03 16:31:59.765551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.765508 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe1369b-d064-40fc-9dd1-47519c1a9807" containerName="revoke-keys" Nov 03 16:31:59.765551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.765524 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bed10d1-6b75-44fe-adaa-0c257a008615" containerName="revoke-keys" Nov 03 16:31:59.769342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.769317 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.815920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.815886 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr"] Nov 03 16:31:59.820008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.819993 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.874615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.874587 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl"] Nov 03 16:31:59.927329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.926863 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr"] Nov 03 16:31:59.940340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940295 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.940509 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940361 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.940509 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940472 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.940654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940562 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.940654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940589 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.940654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940619 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.940821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940675 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrv2\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-kube-api-access-qwrv2\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.940821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940718 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flk6p\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-kube-api-access-flk6p\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.940821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940738 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.940821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940797 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.941025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.940920 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3176a5ac-4339-4bde-b9e4-db18492adf2e-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.941025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.941005 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.941119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.941029 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.941119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.941107 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.941254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.941176 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f27a78-889f-44c6-9e48-54ce4973190a-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.941254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.941225 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f27a78-889f-44c6-9e48-54ce4973190a-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:31:59.941254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.941248 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:31:59.941422 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:31:59.941329 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.006747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.006704 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1678c3-9156-4c06-a700-89520590e7f6" path="/var/lib/kubelet/pods/1b1678c3-9156-4c06-a700-89520590e7f6/volumes" Nov 03 16:32:00.042868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.042780 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3176a5ac-4339-4bde-b9e4-db18492adf2e-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.042868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.042844 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.042874 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.042897 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.042922 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f27a78-889f-44c6-9e48-54ce4973190a-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.042945 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f27a78-889f-44c6-9e48-54ce4973190a-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.042969 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043009 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043048 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043072 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043093 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043127 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043142 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043157 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043187 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrv2\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-kube-api-access-qwrv2\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043256 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk6p\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-kube-api-access-flk6p\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.043584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043299 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.043584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043323 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.044115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043646 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3176a5ac-4339-4bde-b9e4-db18492adf2e-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.044115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043688 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.044115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043687 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f27a78-889f-44c6-9e48-54ce4973190a-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.044115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.043961 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.044382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.044129 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.044382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.044263 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.044382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.044270 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.044382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.044315 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.044596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.044506 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.044657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.044602 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.046031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.046010 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f27a78-889f-44c6-9e48-54ce4973190a-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.046290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.046274 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.046518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.046496 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.046817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.046803 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.105329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.105301 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.105717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.105517 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.105717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.105609 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flk6p\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-kube-api-access-flk6p\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.110543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.110514 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrv2\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-kube-api-access-qwrv2\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.133419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.133395 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:00.293071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.292975 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:00.382603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.382567 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:00.571422 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:00.571391 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3176a5ac_4339_4bde_b9e4_db18492adf2e.slice/crio-44fb84e5cdff2191a12bb71cef0c209f69e475151aa08b4d21e499f563c48f91 WatchSource:0}: Error finding container 44fb84e5cdff2191a12bb71cef0c209f69e475151aa08b4d21e499f563c48f91: Status 404 returned error can't find the container with id 44fb84e5cdff2191a12bb71cef0c209f69e475151aa08b4d21e499f563c48f91 Nov 03 16:32:00.571422 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.571399 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr"] Nov 03 16:32:00.574009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.573978 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:00.574069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.574047 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:00.574109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.574074 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:00.773220 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:00.773175 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f27a78_889f_44c6_9e48_54ce4973190a.slice/crio-34e8b8f1a1e9c0076e1366605d5d58884f9b198ae985d01b4d48c4c55f7cc809 WatchSource:0}: Error finding container 34e8b8f1a1e9c0076e1366605d5d58884f9b198ae985d01b4d48c4c55f7cc809: Status 404 returned error can't find the container with id 34e8b8f1a1e9c0076e1366605d5d58884f9b198ae985d01b4d48c4c55f7cc809 Nov 03 16:32:00.773354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.773277 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl"] Nov 03 16:32:00.775996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.775967 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:00.776067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.776032 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:00.776067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:00.776060 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:01.455739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:01.455683 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" event={"ID":"61f27a78-889f-44c6-9e48-54ce4973190a","Type":"ContainerStarted","Data":"acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d"} Nov 03 16:32:01.455739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:01.455746 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" event={"ID":"61f27a78-889f-44c6-9e48-54ce4973190a","Type":"ContainerStarted","Data":"34e8b8f1a1e9c0076e1366605d5d58884f9b198ae985d01b4d48c4c55f7cc809"} Nov 03 16:32:01.461913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:01.459985 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" event={"ID":"3176a5ac-4339-4bde-b9e4-db18492adf2e","Type":"ContainerStarted","Data":"d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9"} Nov 03 16:32:01.461913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:01.460053 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" event={"ID":"3176a5ac-4339-4bde-b9e4-db18492adf2e","Type":"ContainerStarted","Data":"44fb84e5cdff2191a12bb71cef0c209f69e475151aa08b4d21e499f563c48f91"} Nov 03 16:32:01.582281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:01.582228 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podStartSLOduration=2.582192293 podStartE2EDuration="2.582192293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:01.579394464 +0000 UTC m=+1191.700447284" watchObservedRunningTime="2026-11-03 16:32:01.582192293 +0000 UTC m=+1191.703245186" Nov 03 16:32:01.709538 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:01.709222 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podStartSLOduration=2.709193782 podStartE2EDuration="2.709193782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:01.704894379 +0000 UTC m=+1191.825947200" watchObservedRunningTime="2026-11-03 16:32:01.709193782 +0000 UTC m=+1191.830246604" Nov 03 16:32:02.142234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:02.138599 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:02.150524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:02.149935 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:32:02.150524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:02.150006 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:32:02.382741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:02.382700 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:02.384276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:02.384255 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:32:02.384388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:02.384308 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 16:32:03.086635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:03.086587 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f"] Nov 03 16:32:03.141227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:03.136603 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:32:03.141227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:03.136690 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:32:03.156325 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:03.149046 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b004db6-mz97f"] Nov 03 16:32:03.383854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:03.383762 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:32:03.383854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:03.383829 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 16:32:04.005560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:04.005528 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e88e5a-1f35-49f8-b354-b3f410fef5f0" path="/var/lib/kubelet/pods/23e88e5a-1f35-49f8-b354-b3f410fef5f0/volumes" Nov 03 16:32:04.142653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:04.142615 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:32:04.144237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:04.144182 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:32:04.383158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:04.383063 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:32:04.383158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:04.383121 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 16:32:05.136622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:05.136585 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:32:05.137088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:05.137057 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:32:05.246231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:05.246130 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:05.364231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:05.351947 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:32:05.382864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:05.382833 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:32:05.383033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:05.382881 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 16:32:06.107400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.105416 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj"] Nov 03 16:32:06.158672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.158634 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn"] Nov 03 16:32:06.182706 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.182671 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:32:06.183559 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.183524 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:32:06.185284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.185260 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a7fd45-t7gfj"] Nov 03 16:32:06.230783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.229485 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c6ce0f-6vfjn"] Nov 03 16:32:06.384259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.383631 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:32:06.384259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:06.383739 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 16:32:07.142894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:07.142857 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:32:07.143192 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:07.143150 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:32:07.385735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:07.385699 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:32:07.386364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:07.386313 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 16:32:08.033615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:08.033575 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575a9e39-604f-4f27-a77f-2132e73870d8" path="/var/lib/kubelet/pods/575a9e39-604f-4f27-a77f-2132e73870d8/volumes" Nov 03 16:32:08.034626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:08.034600 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05" path="/var/lib/kubelet/pods/5d7ca9ca-ceaf-46d9-b6fb-c4e6f9dcff05/volumes" Nov 03 16:32:08.141849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:08.141815 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:32:08.142109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:08.142084 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:32:08.217344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:08.217291 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:08.386294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:08.384752 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:32:08.386294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:08.384843 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 16:32:09.151459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.151425 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:09.152086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.152060 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:09.152831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.152808 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:32:09.324742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.324714 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9"] Nov 03 16:32:09.330007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.329991 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.385834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.385805 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:32:09.387505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.387482 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:09.387836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.387752 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:09.388544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.388528 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:32:09.390588 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.390567 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:32:09.390588 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.390580 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:32:09.390711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.390646 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:32:09.430193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.430118 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9"] Nov 03 16:32:09.456791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.456753 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.457023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.457002 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.457146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.457122 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvpfq\" (UniqueName: \"kubernetes.io/projected/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-kube-api-access-lvpfq\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.560185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.560147 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.560511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.560384 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.560511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.560496 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvpfq\" (UniqueName: \"kubernetes.io/projected/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-kube-api-access-lvpfq\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.562077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.561751 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.564311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.564289 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.621853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.621811 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvpfq\" (UniqueName: \"kubernetes.io/projected/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-kube-api-access-lvpfq\") pod \"maas-api-revoke-sub-f7ca2d42-ghws9\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:09.645544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:09.645511 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:10.052184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.052150 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9"] Nov 03 16:32:10.053299 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:10.053272 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c69fdb_6180_4cda_8d9c_05ce011d44cd.slice/crio-7d08981ff884090734c579f3239ae5a2b95284ce2b9d144fc7905f05a37d4b03 WatchSource:0}: Error finding container 7d08981ff884090734c579f3239ae5a2b95284ce2b9d144fc7905f05a37d4b03: Status 404 returned error can't find the container with id 7d08981ff884090734c579f3239ae5a2b95284ce2b9d144fc7905f05a37d4b03 Nov 03 16:32:10.165123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.165084 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:32:10.169177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.169160 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:32:10.197224 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.197178 5946 scope.go:122] "RemoveContainer" containerID="a59735cb94c1349399aa85edc638b61215f003eafbc87c9365fb2a90ecb0ec21" Nov 03 16:32:10.207498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.207479 5946 scope.go:122] "RemoveContainer" containerID="5fa7acebd89368424d490b611be38120ccb0b7801f552ea253c9f2b369ef45ff" Nov 03 16:32:10.567139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.567055 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:10.642178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.642081 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" event={"ID":"b6c69fdb-6180-4cda-8d9c-05ce011d44cd","Type":"ContainerStarted","Data":"bc21b1596942e0b6a037a92808a8e7e2225b7e92772ffea1edccf4a395e4b879"} Nov 03 16:32:10.642370 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.642181 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" event={"ID":"b6c69fdb-6180-4cda-8d9c-05ce011d44cd","Type":"ContainerStarted","Data":"7d08981ff884090734c579f3239ae5a2b95284ce2b9d144fc7905f05a37d4b03"} Nov 03 16:32:10.764282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:10.764226 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" podStartSLOduration=1.7641929140000001 podStartE2EDuration="1.764192914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:10.761646972 +0000 UTC m=+1200.882699798" watchObservedRunningTime="2026-11-03 16:32:10.764192914 +0000 UTC m=+1200.885245737" Nov 03 16:32:11.651605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:11.651543 5946 generic.go:356] "Generic (PLEG): container finished" podID="b6c69fdb-6180-4cda-8d9c-05ce011d44cd" containerID="bc21b1596942e0b6a037a92808a8e7e2225b7e92772ffea1edccf4a395e4b879" exitCode=0 Nov 03 16:32:11.652002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:11.651640 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" event={"ID":"b6c69fdb-6180-4cda-8d9c-05ce011d44cd","Type":"ContainerDied","Data":"bc21b1596942e0b6a037a92808a8e7e2225b7e92772ffea1edccf4a395e4b879"} Nov 03 16:32:12.142636 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.142602 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x"] Nov 03 16:32:12.146839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.146817 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.209922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.209890 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x"] Nov 03 16:32:12.217103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.217074 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8sm\" (UniqueName: \"kubernetes.io/projected/420f1a53-6904-4770-9571-b1dbe9f38248-kube-api-access-dk8sm\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.217271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.217183 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.217359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.217293 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.319187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.319155 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.319399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.319347 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8sm\" (UniqueName: \"kubernetes.io/projected/420f1a53-6904-4770-9571-b1dbe9f38248-kube-api-access-dk8sm\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.319465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.319427 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.320035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.320011 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.321744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.321725 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.381913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.381884 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8sm\" (UniqueName: \"kubernetes.io/projected/420f1a53-6904-4770-9571-b1dbe9f38248-kube-api-access-dk8sm\") pod \"maas-api-revoke-sub-3aa96adc-k9h6x\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.460491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.460409 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:12.589528 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:32:12.589484 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:32:12.804880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.804853 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:12.826970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.826940 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-xks-ca\") pod \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " Nov 03 16:32:12.827114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.827099 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-openshift-ca\") pod \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " Nov 03 16:32:12.827180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.827160 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-kube-api-access-lvpfq\" (UniqueName: \"kubernetes.io/projected/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-kube-api-access-lvpfq\") pod \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\" (UID: \"b6c69fdb-6180-4cda-8d9c-05ce011d44cd\") " Nov 03 16:32:12.827472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.827445 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-openshift-ca" pod "b6c69fdb-6180-4cda-8d9c-05ce011d44cd" (UID: "b6c69fdb-6180-4cda-8d9c-05ce011d44cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:12.827660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.827640 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:12.829162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.829138 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-xks-ca" pod "b6c69fdb-6180-4cda-8d9c-05ce011d44cd" (UID: "b6c69fdb-6180-4cda-8d9c-05ce011d44cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:12.829431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.829412 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-kube-api-access-lvpfq" pod "b6c69fdb-6180-4cda-8d9c-05ce011d44cd" (UID: "b6c69fdb-6180-4cda-8d9c-05ce011d44cd"). InnerVolumeSpecName "kube-api-access-lvpfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:12.846913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.846888 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x"] Nov 03 16:32:12.847066 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:12.847045 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420f1a53_6904_4770_9571_b1dbe9f38248.slice/crio-a4beaab39bb7409e08863ad858c51cd75f4b369fa71ea8d3e14806ea57704eae WatchSource:0}: Error finding container a4beaab39bb7409e08863ad858c51cd75f4b369fa71ea8d3e14806ea57704eae: Status 404 returned error can't find the container with id a4beaab39bb7409e08863ad858c51cd75f4b369fa71ea8d3e14806ea57704eae Nov 03 16:32:12.928892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.928863 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:12.928892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:12.928891 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvpfq\" (UniqueName: \"kubernetes.io/projected/b6c69fdb-6180-4cda-8d9c-05ce011d44cd-kube-api-access-lvpfq\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:13.669542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.669511 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" Nov 03 16:32:13.669542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.669531 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9" event={"ID":"b6c69fdb-6180-4cda-8d9c-05ce011d44cd","Type":"ContainerDied","Data":"7d08981ff884090734c579f3239ae5a2b95284ce2b9d144fc7905f05a37d4b03"} Nov 03 16:32:13.669765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.669586 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d08981ff884090734c579f3239ae5a2b95284ce2b9d144fc7905f05a37d4b03" Nov 03 16:32:13.671236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.671183 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" event={"ID":"420f1a53-6904-4770-9571-b1dbe9f38248","Type":"ContainerStarted","Data":"46f7e97d265f697d823bd21ff8c052a4fc0ffd929c95e69df5f45e0374a732d8"} Nov 03 16:32:13.671360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.671245 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" event={"ID":"420f1a53-6904-4770-9571-b1dbe9f38248","Type":"ContainerStarted","Data":"a4beaab39bb7409e08863ad858c51cd75f4b369fa71ea8d3e14806ea57704eae"} Nov 03 16:32:13.793569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.793504 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" podStartSLOduration=1.793477462 podStartE2EDuration="1.793477462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:13.790910058 +0000 UTC m=+1203.911962880" watchObservedRunningTime="2026-11-03 16:32:13.793477462 +0000 UTC m=+1203.914530286" Nov 03 16:32:13.969110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.969029 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt"] Nov 03 16:32:13.969632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.969616 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b6c69fdb-6180-4cda-8d9c-05ce011d44cd" containerName="revoke-keys" Nov 03 16:32:13.969684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.969633 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c69fdb-6180-4cda-8d9c-05ce011d44cd" containerName="revoke-keys" Nov 03 16:32:13.969744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.969735 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c69fdb-6180-4cda-8d9c-05ce011d44cd" containerName="revoke-keys" Nov 03 16:32:13.973096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:13.973080 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.033263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.033231 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt"] Nov 03 16:32:14.043918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.043883 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.044122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.044098 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbpm\" (UniqueName: \"kubernetes.io/projected/da3ef70d-72b3-4f10-8791-1232d2b2fa94-kube-api-access-jdbpm\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.044272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.044253 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.145770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.145734 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.145964 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.145844 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbpm\" (UniqueName: \"kubernetes.io/projected/da3ef70d-72b3-4f10-8791-1232d2b2fa94-kube-api-access-jdbpm\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.145964 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.145949 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.146606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.146585 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.148582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.148552 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.208131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.208096 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbpm\" (UniqueName: \"kubernetes.io/projected/da3ef70d-72b3-4f10-8791-1232d2b2fa94-kube-api-access-jdbpm\") pod \"maas-api-revoke-sub-1ab4ec34-b4ppt\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.286770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.286742 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:14.601735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.601635 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82"] Nov 03 16:32:14.606894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.606874 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.652714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.652678 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.652871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.652837 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.652936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.652920 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.653014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.652953 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.653014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.652978 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.653014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.653007 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.653136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.653117 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.653200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.653187 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn5h6\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-kube-api-access-zn5h6\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.653256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.653243 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.681070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.681032 5946 generic.go:356] "Generic (PLEG): container finished" podID="420f1a53-6904-4770-9571-b1dbe9f38248" containerID="46f7e97d265f697d823bd21ff8c052a4fc0ffd929c95e69df5f45e0374a732d8" exitCode=0 Nov 03 16:32:14.681272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.681120 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" event={"ID":"420f1a53-6904-4770-9571-b1dbe9f38248","Type":"ContainerDied","Data":"46f7e97d265f697d823bd21ff8c052a4fc0ffd929c95e69df5f45e0374a732d8"} Nov 03 16:32:14.709486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.709456 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82"] Nov 03 16:32:14.724264 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.723988 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt"] Nov 03 16:32:14.754700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754668 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.754865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754752 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.754865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754781 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.754865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754807 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.754865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754836 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754868 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754914 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn5h6\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-kube-api-access-zn5h6\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754948 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.754977 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.755403 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755495 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.755444 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.755501 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.755574 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.755771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.755746 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.758271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.758199 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.758383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.758365 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.816809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.816776 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn5h6\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-kube-api-access-zn5h6\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.817427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.817399 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:14.921159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:14.921079 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:15.363651 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:15.363613 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3431a37_3f66_47e5_bcff_61ba7ebeeb90.slice/crio-abb15647aa663b22c84586f2a258e5418829b4fb46c52633e3a5285555526cda WatchSource:0}: Error finding container abb15647aa663b22c84586f2a258e5418829b4fb46c52633e3a5285555526cda: Status 404 returned error can't find the container with id abb15647aa663b22c84586f2a258e5418829b4fb46c52633e3a5285555526cda Nov 03 16:32:15.365614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.365587 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82"] Nov 03 16:32:15.366876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.366837 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:15.366960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.366928 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:15.367005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.366981 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:15.406249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.402733 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:15.560678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.560641 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln"] Nov 03 16:32:15.565803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.565780 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.623532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.623443 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln"] Nov 03 16:32:15.667076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.667035 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czltb\" (UniqueName: \"kubernetes.io/projected/994743d0-eb54-44cc-a49b-45cf570a96c5-kube-api-access-czltb\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.667282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.667097 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.667399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.667373 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.691497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.691456 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" event={"ID":"e3431a37-3f66-47e5-bcff-61ba7ebeeb90","Type":"ContainerStarted","Data":"9425dcc24f131f0a0aecd8d66d5b0348ee82643ce08e817bfaaaa70df795f301"} Nov 03 16:32:15.691680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.691524 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" event={"ID":"e3431a37-3f66-47e5-bcff-61ba7ebeeb90","Type":"ContainerStarted","Data":"abb15647aa663b22c84586f2a258e5418829b4fb46c52633e3a5285555526cda"} Nov 03 16:32:15.693279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.693241 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" event={"ID":"da3ef70d-72b3-4f10-8791-1232d2b2fa94","Type":"ContainerStarted","Data":"53a9a3e449405c4b5651999f1030306d9965d7f6bdcfe938e02248fcd26789c7"} Nov 03 16:32:15.693279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.693281 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" event={"ID":"da3ef70d-72b3-4f10-8791-1232d2b2fa94","Type":"ContainerStarted","Data":"531e626191ed695fb851b693a27f6cc738046340cde00817a79ee7825db5f519"} Nov 03 16:32:15.768837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.768798 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czltb\" (UniqueName: \"kubernetes.io/projected/994743d0-eb54-44cc-a49b-45cf570a96c5-kube-api-access-czltb\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.769016 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.768855 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.769085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.769044 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.770526 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.770275 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.772835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.772806 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.830565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.830504 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podStartSLOduration=1.83048363 podStartE2EDuration="1.83048363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:15.826729138 +0000 UTC m=+1205.947781963" watchObservedRunningTime="2026-11-03 16:32:15.83048363 +0000 UTC m=+1205.951536454" Nov 03 16:32:15.845889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.845857 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czltb\" (UniqueName: \"kubernetes.io/projected/994743d0-eb54-44cc-a49b-45cf570a96c5-kube-api-access-czltb\") pod \"maas-api-revoke-sub-634928da-68wln\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.852469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.852447 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:15.879693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.879615 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:15.921971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.921845 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:15.967006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.966630 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" podStartSLOduration=2.966604947 podStartE2EDuration="2.966604947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:15.952901273 +0000 UTC m=+1206.073954099" watchObservedRunningTime="2026-11-03 16:32:15.966604947 +0000 UTC m=+1206.087657799" Nov 03 16:32:15.972248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.972162 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-xks-ca\") pod \"420f1a53-6904-4770-9571-b1dbe9f38248\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " Nov 03 16:32:15.972446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.972422 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420f1a53-6904-4770-9571-b1dbe9f38248-kube-api-access-dk8sm\" (UniqueName: \"kubernetes.io/projected/420f1a53-6904-4770-9571-b1dbe9f38248-kube-api-access-dk8sm\") pod \"420f1a53-6904-4770-9571-b1dbe9f38248\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " Nov 03 16:32:15.972546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.972479 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-openshift-ca\") pod \"420f1a53-6904-4770-9571-b1dbe9f38248\" (UID: \"420f1a53-6904-4770-9571-b1dbe9f38248\") " Nov 03 16:32:15.973124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.972898 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-openshift-ca" pod "420f1a53-6904-4770-9571-b1dbe9f38248" (UID: "420f1a53-6904-4770-9571-b1dbe9f38248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:15.973124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.973060 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:15.976292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.976260 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420f1a53-6904-4770-9571-b1dbe9f38248-kube-api-access-dk8sm" pod "420f1a53-6904-4770-9571-b1dbe9f38248" (UID: "420f1a53-6904-4770-9571-b1dbe9f38248"). InnerVolumeSpecName "kube-api-access-dk8sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:15.977429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:15.977410 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-xks-ca" pod "420f1a53-6904-4770-9571-b1dbe9f38248" (UID: "420f1a53-6904-4770-9571-b1dbe9f38248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:16.074929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.074890 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/420f1a53-6904-4770-9571-b1dbe9f38248-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:16.074929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.074930 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8sm\" (UniqueName: \"kubernetes.io/projected/420f1a53-6904-4770-9571-b1dbe9f38248-kube-api-access-dk8sm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:16.274423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.274394 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln"] Nov 03 16:32:16.274874 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:16.274844 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994743d0_eb54_44cc_a49b_45cf570a96c5.slice/crio-5478ea171278a2fe4306c91e8c37f3c82d8d8aaf0bb878b095a7d689711c63fa WatchSource:0}: Error finding container 5478ea171278a2fe4306c91e8c37f3c82d8d8aaf0bb878b095a7d689711c63fa: Status 404 returned error can't find the container with id 5478ea171278a2fe4306c91e8c37f3c82d8d8aaf0bb878b095a7d689711c63fa Nov 03 16:32:16.702895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.702856 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" event={"ID":"994743d0-eb54-44cc-a49b-45cf570a96c5","Type":"ContainerStarted","Data":"d4203528f3ac050ac62177b7e08d27fa07fcdb43753999903f6c1191bf5ac5ad"} Nov 03 16:32:16.703361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.702910 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" event={"ID":"994743d0-eb54-44cc-a49b-45cf570a96c5","Type":"ContainerStarted","Data":"5478ea171278a2fe4306c91e8c37f3c82d8d8aaf0bb878b095a7d689711c63fa"} Nov 03 16:32:16.704353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.704333 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" Nov 03 16:32:16.704475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.704350 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x" event={"ID":"420f1a53-6904-4770-9571-b1dbe9f38248","Type":"ContainerDied","Data":"a4beaab39bb7409e08863ad858c51cd75f4b369fa71ea8d3e14806ea57704eae"} Nov 03 16:32:16.704475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.704396 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4beaab39bb7409e08863ad858c51cd75f4b369fa71ea8d3e14806ea57704eae" Nov 03 16:32:16.706103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.706081 5946 generic.go:356] "Generic (PLEG): container finished" podID="da3ef70d-72b3-4f10-8791-1232d2b2fa94" containerID="53a9a3e449405c4b5651999f1030306d9965d7f6bdcfe938e02248fcd26789c7" exitCode=0 Nov 03 16:32:16.706239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.706123 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" event={"ID":"da3ef70d-72b3-4f10-8791-1232d2b2fa94","Type":"ContainerDied","Data":"53a9a3e449405c4b5651999f1030306d9965d7f6bdcfe938e02248fcd26789c7"} Nov 03 16:32:16.840916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.840856 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" podStartSLOduration=1.84083839 podStartE2EDuration="1.84083839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:16.82550733 +0000 UTC m=+1206.946560133" watchObservedRunningTime="2026-11-03 16:32:16.84083839 +0000 UTC m=+1206.961891213" Nov 03 16:32:16.938334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.935325 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:32:16.938334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:16.935388 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:32:17.740471 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:17.740434 5946 generic.go:356] "Generic (PLEG): container finished" podID="994743d0-eb54-44cc-a49b-45cf570a96c5" containerID="d4203528f3ac050ac62177b7e08d27fa07fcdb43753999903f6c1191bf5ac5ad" exitCode=0 Nov 03 16:32:17.740977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:17.740529 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" event={"ID":"994743d0-eb54-44cc-a49b-45cf570a96c5","Type":"ContainerDied","Data":"d4203528f3ac050ac62177b7e08d27fa07fcdb43753999903f6c1191bf5ac5ad"} Nov 03 16:32:17.940382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:17.936639 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:32:17.940382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:17.936697 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:32:17.941985 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:17.941621 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:17.999893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:17.999823 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-openshift-ca\") pod \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " Nov 03 16:32:18.000033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.000015 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3ef70d-72b3-4f10-8791-1232d2b2fa94-kube-api-access-jdbpm\" (UniqueName: \"kubernetes.io/projected/da3ef70d-72b3-4f10-8791-1232d2b2fa94-kube-api-access-jdbpm\") pod \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " Nov 03 16:32:18.000140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.000124 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-xks-ca\") pod \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\" (UID: \"da3ef70d-72b3-4f10-8791-1232d2b2fa94\") " Nov 03 16:32:18.000195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.000178 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-openshift-ca" pod "da3ef70d-72b3-4f10-8791-1232d2b2fa94" (UID: "da3ef70d-72b3-4f10-8791-1232d2b2fa94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:18.000565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.000544 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:18.002187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.002163 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3ef70d-72b3-4f10-8791-1232d2b2fa94-kube-api-access-jdbpm" pod "da3ef70d-72b3-4f10-8791-1232d2b2fa94" (UID: "da3ef70d-72b3-4f10-8791-1232d2b2fa94"). InnerVolumeSpecName "kube-api-access-jdbpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:18.002337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.002304 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-xks-ca" pod "da3ef70d-72b3-4f10-8791-1232d2b2fa94" (UID: "da3ef70d-72b3-4f10-8791-1232d2b2fa94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:18.102073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.102031 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbpm\" (UniqueName: \"kubernetes.io/projected/da3ef70d-72b3-4f10-8791-1232d2b2fa94-kube-api-access-jdbpm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:18.102073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.102072 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ef70d-72b3-4f10-8791-1232d2b2fa94-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:18.751069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.751028 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" Nov 03 16:32:18.751069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.751054 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt" event={"ID":"da3ef70d-72b3-4f10-8791-1232d2b2fa94","Type":"ContainerDied","Data":"531e626191ed695fb851b693a27f6cc738046340cde00817a79ee7825db5f519"} Nov 03 16:32:18.751569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.751118 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="531e626191ed695fb851b693a27f6cc738046340cde00817a79ee7825db5f519" Nov 03 16:32:18.938381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.938350 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:32:18.938671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.938618 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:32:18.943090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:18.943069 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:19.011769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.011694 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-openshift-ca\") pod \"994743d0-eb54-44cc-a49b-45cf570a96c5\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " Nov 03 16:32:19.011900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.011851 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994743d0-eb54-44cc-a49b-45cf570a96c5-kube-api-access-czltb\" (UniqueName: \"kubernetes.io/projected/994743d0-eb54-44cc-a49b-45cf570a96c5-kube-api-access-czltb\") pod \"994743d0-eb54-44cc-a49b-45cf570a96c5\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " Nov 03 16:32:19.011963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.011953 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-xks-ca\") pod \"994743d0-eb54-44cc-a49b-45cf570a96c5\" (UID: \"994743d0-eb54-44cc-a49b-45cf570a96c5\") " Nov 03 16:32:19.012100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.012072 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-openshift-ca" pod "994743d0-eb54-44cc-a49b-45cf570a96c5" (UID: "994743d0-eb54-44cc-a49b-45cf570a96c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:19.012368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.012353 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:19.039433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.033328 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-xks-ca" pod "994743d0-eb54-44cc-a49b-45cf570a96c5" (UID: "994743d0-eb54-44cc-a49b-45cf570a96c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:19.041867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.041834 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994743d0-eb54-44cc-a49b-45cf570a96c5-kube-api-access-czltb" pod "994743d0-eb54-44cc-a49b-45cf570a96c5" (UID: "994743d0-eb54-44cc-a49b-45cf570a96c5"). InnerVolumeSpecName "kube-api-access-czltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:19.131234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.113941 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994743d0-eb54-44cc-a49b-45cf570a96c5-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:19.131234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.113981 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czltb\" (UniqueName: \"kubernetes.io/projected/994743d0-eb54-44cc-a49b-45cf570a96c5-kube-api-access-czltb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:19.759540 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.759501 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" Nov 03 16:32:19.759977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.759534 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln" event={"ID":"994743d0-eb54-44cc-a49b-45cf570a96c5","Type":"ContainerDied","Data":"5478ea171278a2fe4306c91e8c37f3c82d8d8aaf0bb878b095a7d689711c63fa"} Nov 03 16:32:19.759977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.759618 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5478ea171278a2fe4306c91e8c37f3c82d8d8aaf0bb878b095a7d689711c63fa" Nov 03 16:32:19.938671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.938624 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:32:19.938838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:19.938706 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:32:20.063153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:20.062998 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx"] Nov 03 16:32:20.069672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:20.069640 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478e8921-s48sx"] Nov 03 16:32:20.211880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:20.211846 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:20.932467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:20.932427 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:32:20.933073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:20.933038 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:32:21.682623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:21.682588 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4"] Nov 03 16:32:21.682805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:21.682784 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" containerID="cri-o://966698430d884b2264b06d9a4c3a494c3fc3f0d79ae4785b4ee089b62a5ad25b" gracePeriod=30 Nov 03 16:32:21.733730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:21.733696 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg"] Nov 03 16:32:21.733959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:21.733935 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" containerID="cri-o://fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e" gracePeriod=30 Nov 03 16:32:21.921866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:21.921832 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:32:21.922036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:21.921889 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:32:22.005639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:22.005608 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184c6de3-969d-4432-bd5d-62dc8144dd81" path="/var/lib/kubelet/pods/184c6de3-969d-4432-bd5d-62dc8144dd81/volumes" Nov 03 16:32:22.063363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:22.063323 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6"] Nov 03 16:32:22.070639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:22.070603 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c37c1961-mgsh6"] Nov 03 16:32:22.926542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:22.926508 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:22.926911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:22.926892 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:22.927529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:22.927508 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:32:24.004596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:24.004561 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1c4ff8-70b5-40c9-80c2-009b8afc7933" path="/var/lib/kubelet/pods/1b1c4ff8-70b5-40c9-80c2-009b8afc7933/volumes" Nov 03 16:32:24.074177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:24.074144 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz"] Nov 03 16:32:24.126824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:24.126791 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc8fd95-jt6kz"] Nov 03 16:32:24.946656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:24.946626 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:25.064081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.064023 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q"] Nov 03 16:32:25.114901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.114865 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eeb0724-jbc4q"] Nov 03 16:32:25.555576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.555538 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb"] Nov 03 16:32:25.556130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556117 5946 cpu_manager.go:414] "containerMap: removing container" podUID="420f1a53-6904-4770-9571-b1dbe9f38248" containerName="revoke-keys" Nov 03 16:32:25.556171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556133 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420f1a53-6904-4770-9571-b1dbe9f38248" containerName="revoke-keys" Nov 03 16:32:25.556171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556143 5946 cpu_manager.go:414] "containerMap: removing container" podUID="da3ef70d-72b3-4f10-8791-1232d2b2fa94" containerName="revoke-keys" Nov 03 16:32:25.556171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556148 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3ef70d-72b3-4f10-8791-1232d2b2fa94" containerName="revoke-keys" Nov 03 16:32:25.556171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556170 5946 cpu_manager.go:414] "containerMap: removing container" podUID="994743d0-eb54-44cc-a49b-45cf570a96c5" containerName="revoke-keys" Nov 03 16:32:25.556350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556176 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994743d0-eb54-44cc-a49b-45cf570a96c5" containerName="revoke-keys" Nov 03 16:32:25.556350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556274 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="420f1a53-6904-4770-9571-b1dbe9f38248" containerName="revoke-keys" Nov 03 16:32:25.556350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556283 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3ef70d-72b3-4f10-8791-1232d2b2fa94" containerName="revoke-keys" Nov 03 16:32:25.556350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.556292 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="994743d0-eb54-44cc-a49b-45cf570a96c5" containerName="revoke-keys" Nov 03 16:32:25.561150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.561129 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.622694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.622660 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb"] Nov 03 16:32:25.667858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.667823 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:32:25.667858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.667851 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:32:25.668084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.667881 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:32:25.668084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.667932 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:32:25.722169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.722134 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.722396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.722188 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.722460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.722415 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kqh\" (UniqueName: \"kubernetes.io/projected/b091a762-284d-4410-9a11-089dd21e0b70-kube-api-access-f7kqh\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.823830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.823750 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.823830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.823794 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.824014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.823887 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kqh\" (UniqueName: \"kubernetes.io/projected/b091a762-284d-4410-9a11-089dd21e0b70-kube-api-access-f7kqh\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.824451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.824430 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.826369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.826352 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:25.884923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:25.884895 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kqh\" (UniqueName: \"kubernetes.io/projected/b091a762-284d-4410-9a11-089dd21e0b70-kube-api-access-f7kqh\") pod \"maas-api-revoke-sub-1daea440-q56nb\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:26.004655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:26.004622 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9f0a0b-91ae-4d58-b97e-53a436972c3e" path="/var/lib/kubelet/pods/3c9f0a0b-91ae-4d58-b97e-53a436972c3e/volumes" Nov 03 16:32:26.005100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:26.005086 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e94dafd-fee1-4bf6-b654-1c2ff9054d91" path="/var/lib/kubelet/pods/3e94dafd-fee1-4bf6-b654-1c2ff9054d91/volumes" Nov 03 16:32:26.175767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:26.175697 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:26.566271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:26.566235 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb"] Nov 03 16:32:26.566695 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:26.566672 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb091a762_284d_4410_9a11_089dd21e0b70.slice/crio-0affe522950b850fb657f68ac357ce33daf9ae857433a00e74d1641a5f86aae9 WatchSource:0}: Error finding container 0affe522950b850fb657f68ac357ce33daf9ae857433a00e74d1641a5f86aae9: Status 404 returned error can't find the container with id 0affe522950b850fb657f68ac357ce33daf9ae857433a00e74d1641a5f86aae9 Nov 03 16:32:26.818406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:26.818368 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" event={"ID":"b091a762-284d-4410-9a11-089dd21e0b70","Type":"ContainerStarted","Data":"0affe522950b850fb657f68ac357ce33daf9ae857433a00e74d1641a5f86aae9"} Nov 03 16:32:26.820172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:26.820147 5946 generic.go:356] "Generic (PLEG): container finished" podID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerID="966698430d884b2264b06d9a4c3a494c3fc3f0d79ae4785b4ee089b62a5ad25b" exitCode=0 Nov 03 16:32:26.820306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:26.820232 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" event={"ID":"56316bb2-7f63-4700-a987-96d2ca2cb2fe","Type":"ContainerDied","Data":"966698430d884b2264b06d9a4c3a494c3fc3f0d79ae4785b4ee089b62a5ad25b"} Nov 03 16:32:27.123973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.123949 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:32:27.163648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.163619 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.243515 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-data\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.243646 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-kube-api-access-5w4d8\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-kube-api-access-5w4d8\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.243698 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-envoy\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.243740 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-token\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-token\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.243834 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istiod-ca-cert\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.243879 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-credential-socket\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.243948 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-certs\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.244010 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-socket\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.247232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.244052 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-podinfo\") pod \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\" (UID: \"56316bb2-7f63-4700-a987-96d2ca2cb2fe\") " Nov 03 16:32:27.251543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.250803 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-data" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.251543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.251148 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-certs" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.251543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.251369 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-socket" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.252176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.252130 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-credential-socket" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.254760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.254316 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istiod-ca-cert" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:27.269363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.269125 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-token" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:27.270361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.269875 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-podinfo" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:32:27.270620 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.270285 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-kube-api-access-5w4d8" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "kube-api-access-5w4d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:27.272826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.272788 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-envoy" pod "56316bb2-7f63-4700-a987-96d2ca2cb2fe" (UID: "56316bb2-7f63-4700-a987-96d2ca2cb2fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.350293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.349488 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-envoy\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.350293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.349562 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-socket\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.350293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.349633 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b6d0289-081e-4869-81fd-6c2a2b3621df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d0289-081e-4869-81fd-6c2a2b3621df-istiod-ca-cert\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.350293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.349714 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-data\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.350293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.349760 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-certs\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.350293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.349829 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-podinfo\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.352249 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6d0289-081e-4869-81fd-6c2a2b3621df-istiod-ca-cert" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.352612 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-token\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.352736 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-data" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.352815 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-kube-api-access-k6q92\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-kube-api-access-k6q92\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.352890 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-credential-socket\") pod \"9b6d0289-081e-4869-81fd-6c2a2b3621df\" (UID: \"9b6d0289-081e-4869-81fd-6c2a2b3621df\") " Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353009 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-certs" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353665 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353702 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4d8\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-kube-api-access-5w4d8\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353719 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353733 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353747 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6d0289-081e-4869-81fd-6c2a2b3621df-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353775 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353789 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353805 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353818 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353847 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353863 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56316bb2-7f63-4700-a987-96d2ca2cb2fe-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.353877 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56316bb2-7f63-4700-a987-96d2ca2cb2fe-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.354117 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-credential-socket" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.354991 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-socket" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.358267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.357949 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-kube-api-access-k6q92" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "kube-api-access-k6q92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:27.360197 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.360168 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-podinfo" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:32:27.362152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.360605 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-envoy" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:32:27.362909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.362875 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-token" pod "9b6d0289-081e-4869-81fd-6c2a2b3621df" (UID: "9b6d0289-081e-4869-81fd-6c2a2b3621df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:27.455821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.455738 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.455821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.455776 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6q92\" (UniqueName: \"kubernetes.io/projected/9b6d0289-081e-4869-81fd-6c2a2b3621df-kube-api-access-k6q92\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.455821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.455792 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.455821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.455806 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.455821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.455818 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6d0289-081e-4869-81fd-6c2a2b3621df-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.455821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.455830 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6d0289-081e-4869-81fd-6c2a2b3621df-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:27.850626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.839889 5946 generic.go:356] "Generic (PLEG): container finished" podID="b091a762-284d-4410-9a11-089dd21e0b70" containerID="bf90f5623c08d09d8d1b06d5f1228ab0818f708366489f3285c46eb1ab722c62" exitCode=0 Nov 03 16:32:27.850626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.840055 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" event={"ID":"b091a762-284d-4410-9a11-089dd21e0b70","Type":"ContainerDied","Data":"bf90f5623c08d09d8d1b06d5f1228ab0818f708366489f3285c46eb1ab722c62"} Nov 03 16:32:27.850626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.842832 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" event={"ID":"56316bb2-7f63-4700-a987-96d2ca2cb2fe","Type":"ContainerDied","Data":"a83af99c88e7f38adc9a8f461a28797707dc66d9ce6caf97da0a87308e6be9a3"} Nov 03 16:32:27.850626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.842915 5946 scope.go:122] "RemoveContainer" containerID="966698430d884b2264b06d9a4c3a494c3fc3f0d79ae4785b4ee089b62a5ad25b" Nov 03 16:32:27.850626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.843132 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4" Nov 03 16:32:27.855454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.855420 5946 generic.go:356] "Generic (PLEG): container finished" podID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerID="fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e" exitCode=0 Nov 03 16:32:27.855719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.855697 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" event={"ID":"9b6d0289-081e-4869-81fd-6c2a2b3621df","Type":"ContainerDied","Data":"fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e"} Nov 03 16:32:27.856540 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.856517 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" event={"ID":"9b6d0289-081e-4869-81fd-6c2a2b3621df","Type":"ContainerDied","Data":"6516b33fb72e28d588d1833ee089c46278d50a4cac5c427c1544fdf1175513c1"} Nov 03 16:32:27.856843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.856060 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg" Nov 03 16:32:27.888435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.888406 5946 scope.go:122] "RemoveContainer" containerID="fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e" Nov 03 16:32:27.906658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.906347 5946 scope.go:122] "RemoveContainer" containerID="fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e" Nov 03 16:32:27.907035 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:32:27.906936 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e\": container with ID starting with fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e not found: ID does not exist" containerID="fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e" Nov 03 16:32:27.907035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:27.906974 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e"} err="failed to get container status \"fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e\": rpc error: code = NotFound desc = could not find container \"fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e\": container with ID starting with fcf2e110311f6e3011b27c252d2978bd8b32ac001bcbf08b733b7f3b1465721e not found: ID does not exist" Nov 03 16:32:28.125333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:28.124310 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4"] Nov 03 16:32:28.149233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:28.148645 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qpmp4"] Nov 03 16:32:28.244231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:28.243306 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg"] Nov 03 16:32:28.268915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:28.268873 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ds4jg"] Nov 03 16:32:29.318398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.318346 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:29.517250 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.495736 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:29.543324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.541469 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:29.550929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.550894 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:29.727223 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.724322 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 16:32:29.797476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.763315 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:29.798131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.798090 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:29.799630 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.799599 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:32:29.956332 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.956293 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" event={"ID":"b091a762-284d-4410-9a11-089dd21e0b70","Type":"ContainerDied","Data":"0affe522950b850fb657f68ac357ce33daf9ae857433a00e74d1641a5f86aae9"} Nov 03 16:32:29.956596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:29.956577 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0affe522950b850fb657f68ac357ce33daf9ae857433a00e74d1641a5f86aae9" Nov 03 16:32:30.089541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.083357 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 16:32:30.089541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.086292 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:30.187233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.147023 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" path="/var/lib/kubelet/pods/56316bb2-7f63-4700-a987-96d2ca2cb2fe/volumes" Nov 03 16:32:30.211237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.188379 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" path="/var/lib/kubelet/pods/9b6d0289-081e-4869-81fd-6c2a2b3621df/volumes" Nov 03 16:32:30.211990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.211959 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl"] Nov 03 16:32:30.212156 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.157739 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:30.238282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.221134 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c10330-94ldl"] Nov 03 16:32:30.323232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.320560 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:30.323232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.322008 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:32:30.323232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.323229 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b091a762-284d-4410-9a11-089dd21e0b70-kube-api-access-f7kqh\" (UniqueName: \"kubernetes.io/projected/b091a762-284d-4410-9a11-089dd21e0b70-kube-api-access-f7kqh\") pod \"b091a762-284d-4410-9a11-089dd21e0b70\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " Nov 03 16:32:30.323843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.323649 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-openshift-ca\") pod \"b091a762-284d-4410-9a11-089dd21e0b70\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " Nov 03 16:32:30.323843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.323780 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-xks-ca\") pod \"b091a762-284d-4410-9a11-089dd21e0b70\" (UID: \"b091a762-284d-4410-9a11-089dd21e0b70\") " Nov 03 16:32:30.351281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.324424 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-openshift-ca" pod "b091a762-284d-4410-9a11-089dd21e0b70" (UID: "b091a762-284d-4410-9a11-089dd21e0b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:30.351281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.328535 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:30.370461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.370378 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-xks-ca" pod "b091a762-284d-4410-9a11-089dd21e0b70" (UID: "b091a762-284d-4410-9a11-089dd21e0b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:30.401123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.400972 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b091a762-284d-4410-9a11-089dd21e0b70-kube-api-access-f7kqh" pod "b091a762-284d-4410-9a11-089dd21e0b70" (UID: "b091a762-284d-4410-9a11-089dd21e0b70"). InnerVolumeSpecName "kube-api-access-f7kqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:30.456231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.433589 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091a762-284d-4410-9a11-089dd21e0b70-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:30.456560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.456523 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7kqh\" (UniqueName: \"kubernetes.io/projected/b091a762-284d-4410-9a11-089dd21e0b70-kube-api-access-f7kqh\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:30.485221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.484679 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:30.510644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.510565 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:32:30.516976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.512411 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:32:30.586351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.584555 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 16:32:30.589775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.589745 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:32:30.625479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.625387 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:32:30.629641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.629618 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:32:30.631576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.631548 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:32:30.965270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:30.965151 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb" Nov 03 16:32:31.062787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.062752 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59"] Nov 03 16:32:31.063450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063431 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" Nov 03 16:32:31.063524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063453 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" Nov 03 16:32:31.063524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063483 5946 cpu_manager.go:414] "containerMap: removing container" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" Nov 03 16:32:31.063524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063489 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" Nov 03 16:32:31.063524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063507 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b091a762-284d-4410-9a11-089dd21e0b70" containerName="revoke-keys" Nov 03 16:32:31.063524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063512 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b091a762-284d-4410-9a11-089dd21e0b70" containerName="revoke-keys" Nov 03 16:32:31.063751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063602 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b091a762-284d-4410-9a11-089dd21e0b70" containerName="revoke-keys" Nov 03 16:32:31.063751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063621 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6d0289-081e-4869-81fd-6c2a2b3621df" containerName="istio-proxy" Nov 03 16:32:31.063751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.063631 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="56316bb2-7f63-4700-a987-96d2ca2cb2fe" containerName="istio-proxy" Nov 03 16:32:31.067975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.067958 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.128657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.128623 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59"] Nov 03 16:32:31.176433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.176399 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:31.182915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.182882 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.182947 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.183059 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.183181 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.183244 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.183392 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.183456 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.183519 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.183642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.183615 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mjn\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-kube-api-access-m6mjn\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285090 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285152 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mjn\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-kube-api-access-m6mjn\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285235 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285270 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285309 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285380 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285418 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285504 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285558 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.285805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.285656 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.286250 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.286191 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.286527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.286506 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.286615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.286546 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.286768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.286741 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.288277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.288185 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.289063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.289042 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.355382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.355346 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.361329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.361298 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mjn\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-kube-api-access-m6mjn\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.373001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.372970 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4"] Nov 03 16:32:31.379344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.379319 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.385949 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.385906 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:31.441564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.441518 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4"] Nov 03 16:32:31.488713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.488660 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.488899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.488752 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.488899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.488782 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40910515-e3aa-4382-8139-2fb17a68649f-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.488899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.488826 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40910515-e3aa-4382-8139-2fb17a68649f-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.488899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.488857 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh7fp\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-kube-api-access-hh7fp\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.489151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.488976 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.489151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.489023 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.489151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.489048 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.489350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.489151 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590160 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590263 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590303 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590323 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40910515-e3aa-4382-8139-2fb17a68649f-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590455 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40910515-e3aa-4382-8139-2fb17a68649f-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590483 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh7fp\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-kube-api-access-hh7fp\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590573 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590616 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590867 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.590942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590899 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.591105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.590943 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.591105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.591043 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.591303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.591224 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40910515-e3aa-4382-8139-2fb17a68649f-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.591385 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.591311 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.593569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.593545 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.593662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.593610 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40910515-e3aa-4382-8139-2fb17a68649f-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.651154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.651117 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.651368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.651338 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh7fp\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-kube-api-access-hh7fp\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.694695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.694666 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:31.829904 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.829877 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59"] Nov 03 16:32:31.830012 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:31.829956 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec3c61f_ca5f_4bb5_af09_8885570c37e8.slice/crio-b7c81ea42ef25498530ff10e297c1816b90a1762c5bcb9f77ec773807686615a WatchSource:0}: Error finding container b7c81ea42ef25498530ff10e297c1816b90a1762c5bcb9f77ec773807686615a: Status 404 returned error can't find the container with id b7c81ea42ef25498530ff10e297c1816b90a1762c5bcb9f77ec773807686615a Nov 03 16:32:31.832646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.832615 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:31.832746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.832677 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:31.832746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.832707 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:31.975882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.975846 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" event={"ID":"1ec3c61f-ca5f-4bb5-af09-8885570c37e8","Type":"ContainerStarted","Data":"9f598bc7fe8791d40da0306afc9a9d7a210b5e5b4f47e2da6f89965c45584d1d"} Nov 03 16:32:31.976047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:31.975911 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" event={"ID":"1ec3c61f-ca5f-4bb5-af09-8885570c37e8","Type":"ContainerStarted","Data":"b7c81ea42ef25498530ff10e297c1816b90a1762c5bcb9f77ec773807686615a"} Nov 03 16:32:32.008897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.008865 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750a6935-778a-4b2f-b049-a9e07ec19b03" path="/var/lib/kubelet/pods/750a6935-778a-4b2f-b049-a9e07ec19b03/volumes" Nov 03 16:32:32.065252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.063662 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x"] Nov 03 16:32:32.071200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.071174 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a1c070-qxl6x"] Nov 03 16:32:32.119868 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:32.119831 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40910515_e3aa_4382_8139_2fb17a68649f.slice/crio-0195fc3a67cf618cd560307c5664c70f462352c705334abef57c7dc1885e77e7 WatchSource:0}: Error finding container 0195fc3a67cf618cd560307c5664c70f462352c705334abef57c7dc1885e77e7: Status 404 returned error can't find the container with id 0195fc3a67cf618cd560307c5664c70f462352c705334abef57c7dc1885e77e7 Nov 03 16:32:32.121413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.121197 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podStartSLOduration=1.121178101 podStartE2EDuration="1.121178101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:32.103162903 +0000 UTC m=+1222.224215728" watchObservedRunningTime="2026-11-03 16:32:32.121178101 +0000 UTC m=+1222.242230926" Nov 03 16:32:32.123188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.123154 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:32.123339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.123325 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:32.123401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.123359 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:32.126875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.126855 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4"] Nov 03 16:32:32.386584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.386487 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:32.387994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.387968 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:32:32.388062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.388029 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:32:32.986376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.986312 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" event={"ID":"40910515-e3aa-4382-8139-2fb17a68649f","Type":"ContainerStarted","Data":"42154addac6e7a0815008ff21cec274dcba5ac2d33ba5dea418d8d425420ce27"} Nov 03 16:32:32.986742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:32.986708 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" event={"ID":"40910515-e3aa-4382-8139-2fb17a68649f","Type":"ContainerStarted","Data":"0195fc3a67cf618cd560307c5664c70f462352c705334abef57c7dc1885e77e7"} Nov 03 16:32:33.164941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.164873 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podStartSLOduration=2.164851382 podStartE2EDuration="2.164851382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:33.164674594 +0000 UTC m=+1223.285727419" watchObservedRunningTime="2026-11-03 16:32:33.164851382 +0000 UTC m=+1223.285904209" Nov 03 16:32:33.386704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.386613 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:32:33.386704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.386668 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:32:33.695678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.695603 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:33.697006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.696968 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 16:32:33.697155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.697022 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 16:32:33.969329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.969145 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf"] Nov 03 16:32:33.975478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.975452 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:33.994003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.993976 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:33.994144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.994010 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4m7n\" (UniqueName: \"kubernetes.io/projected/1bc142a9-f3c9-408b-9b51-10578efd1196-kube-api-access-r4m7n\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:33.994144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:33.994028 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.004453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.004014 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c48378f-a619-4805-8685-7acec80579e8" path="/var/lib/kubelet/pods/6c48378f-a619-4805-8685-7acec80579e8/volumes" Nov 03 16:32:34.008535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.008507 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf"] Nov 03 16:32:34.046418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.046379 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:32:34.047975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.046736 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:32:34.051497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.049878 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:32:34.051497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.049995 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:32:34.095689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.095654 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.095689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.095693 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4m7n\" (UniqueName: \"kubernetes.io/projected/1bc142a9-f3c9-408b-9b51-10578efd1196-kube-api-access-r4m7n\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.095893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.095711 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.097025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.096998 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.099593 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.099572 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.169952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.169883 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4m7n\" (UniqueName: \"kubernetes.io/projected/1bc142a9-f3c9-408b-9b51-10578efd1196-kube-api-access-r4m7n\") pod \"maas-api-revoke-sub-2351bf03-ppdcf\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.291198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.291166 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:34.387096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.387049 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:32:34.387608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.387114 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:32:34.695448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.695364 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 16:32:34.695448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.695418 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 16:32:34.771426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:34.771388 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf"] Nov 03 16:32:34.774756 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:34.774723 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc142a9_f3c9_408b_9b51_10578efd1196.slice/crio-e9d52d6a5f4d41104c01ca113849d7e88d263b29bec6fdce4f7a683630f3ec16 WatchSource:0}: Error finding container e9d52d6a5f4d41104c01ca113849d7e88d263b29bec6fdce4f7a683630f3ec16: Status 404 returned error can't find the container with id e9d52d6a5f4d41104c01ca113849d7e88d263b29bec6fdce4f7a683630f3ec16 Nov 03 16:32:35.003899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.003871 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" event={"ID":"1bc142a9-f3c9-408b-9b51-10578efd1196","Type":"ContainerStarted","Data":"e9d52d6a5f4d41104c01ca113849d7e88d263b29bec6fdce4f7a683630f3ec16"} Nov 03 16:32:35.070266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.070019 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq"] Nov 03 16:32:35.074246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.074199 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c520da9-vbzzq"] Nov 03 16:32:35.386767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.386737 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:32:35.386974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.386851 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:32:35.401042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.401014 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:35.695930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.695815 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 16:32:35.695930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:35.695898 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 16:32:36.005767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.005734 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7725f188-5cf8-44bf-a684-d78b97aef0a4" path="/var/lib/kubelet/pods/7725f188-5cf8-44bf-a684-d78b97aef0a4/volumes" Nov 03 16:32:36.072873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.071893 5946 generic.go:356] "Generic (PLEG): container finished" podID="1bc142a9-f3c9-408b-9b51-10578efd1196" containerID="846c25e022bc4c7c500fceef22ae39193c831933fbac235b1018b5820bb82ef6" exitCode=0 Nov 03 16:32:36.072873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.072084 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" event={"ID":"1bc142a9-f3c9-408b-9b51-10578efd1196","Type":"ContainerDied","Data":"846c25e022bc4c7c500fceef22ae39193c831933fbac235b1018b5820bb82ef6"} Nov 03 16:32:36.072873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.072136 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs"] Nov 03 16:32:36.075704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.075679 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb00395-6gbqs"] Nov 03 16:32:36.386604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.386562 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:32:36.386788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.386625 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:32:36.695333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.695248 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 16:32:36.695333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:36.695302 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 16:32:37.369889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.369758 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:37.387037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.387005 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:32:37.387234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.387057 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:32:37.496551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.496522 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc142a9-f3c9-408b-9b51-10578efd1196-kube-api-access-r4m7n\" (UniqueName: \"kubernetes.io/projected/1bc142a9-f3c9-408b-9b51-10578efd1196-kube-api-access-r4m7n\") pod \"1bc142a9-f3c9-408b-9b51-10578efd1196\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " Nov 03 16:32:37.496754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.496559 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-openshift-ca\") pod \"1bc142a9-f3c9-408b-9b51-10578efd1196\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " Nov 03 16:32:37.496754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.496593 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-xks-ca\") pod \"1bc142a9-f3c9-408b-9b51-10578efd1196\" (UID: \"1bc142a9-f3c9-408b-9b51-10578efd1196\") " Nov 03 16:32:37.497254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.496942 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-openshift-ca" pod "1bc142a9-f3c9-408b-9b51-10578efd1196" (UID: "1bc142a9-f3c9-408b-9b51-10578efd1196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:37.497254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.497231 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:37.498900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.498870 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-xks-ca" pod "1bc142a9-f3c9-408b-9b51-10578efd1196" (UID: "1bc142a9-f3c9-408b-9b51-10578efd1196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:37.499027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.499002 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc142a9-f3c9-408b-9b51-10578efd1196-kube-api-access-r4m7n" pod "1bc142a9-f3c9-408b-9b51-10578efd1196" (UID: "1bc142a9-f3c9-408b-9b51-10578efd1196"). InnerVolumeSpecName "kube-api-access-r4m7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:37.598436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.598400 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4m7n\" (UniqueName: \"kubernetes.io/projected/1bc142a9-f3c9-408b-9b51-10578efd1196-kube-api-access-r4m7n\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:37.598436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.598435 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc142a9-f3c9-408b-9b51-10578efd1196-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:37.695278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.695172 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 16:32:37.695278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:37.695254 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 16:32:38.004112 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.004068 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf26191-7e04-4187-9c38-f2f16ad202f8" path="/var/lib/kubelet/pods/7cf26191-7e04-4187-9c38-f2f16ad202f8/volumes" Nov 03 16:32:38.092091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.092056 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" Nov 03 16:32:38.092770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.092065 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf" event={"ID":"1bc142a9-f3c9-408b-9b51-10578efd1196","Type":"ContainerDied","Data":"e9d52d6a5f4d41104c01ca113849d7e88d263b29bec6fdce4f7a683630f3ec16"} Nov 03 16:32:38.092852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.092795 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d52d6a5f4d41104c01ca113849d7e88d263b29bec6fdce4f7a683630f3ec16" Nov 03 16:32:38.172503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.172466 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6"] Nov 03 16:32:38.173059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.173044 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1bc142a9-f3c9-408b-9b51-10578efd1196" containerName="revoke-keys" Nov 03 16:32:38.173123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.173061 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc142a9-f3c9-408b-9b51-10578efd1196" containerName="revoke-keys" Nov 03 16:32:38.173174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.173164 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc142a9-f3c9-408b-9b51-10578efd1196" containerName="revoke-keys" Nov 03 16:32:38.177024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.177001 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.236233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.235554 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:32:38.236436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.236314 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:32:38.236436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.236406 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:32:38.236541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.236454 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:32:38.272868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.272759 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd"] Nov 03 16:32:38.277260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.277231 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.280733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.279296 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6"] Nov 03 16:32:38.308555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.308528 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.308709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.308669 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.308755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.308710 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6qcw\" (UniqueName: \"kubernetes.io/projected/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-kube-api-access-g6qcw\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.353668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.353346 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd"] Nov 03 16:32:38.387050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.387017 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:32:38.387254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.387071 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:32:38.410185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.410155 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.410404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.410194 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.410404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.410384 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lvn\" (UniqueName: \"kubernetes.io/projected/2ad2dae2-3cce-4572-94bc-8f67b2837c98-kube-api-access-p2lvn\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.410477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.410434 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.410595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.410568 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.410725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.410599 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6qcw\" (UniqueName: \"kubernetes.io/projected/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-kube-api-access-g6qcw\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.411270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.411250 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.413144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.413122 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.472174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.472145 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6qcw\" (UniqueName: \"kubernetes.io/projected/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-kube-api-access-g6qcw\") pod \"maas-api-revoke-sub-00940418-jxxr6\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.491938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.491905 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:38.513650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.513619 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.513796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.513674 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.513867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.513810 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lvn\" (UniqueName: \"kubernetes.io/projected/2ad2dae2-3cce-4572-94bc-8f67b2837c98-kube-api-access-p2lvn\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.514728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.514705 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.516972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.516946 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.591561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.584619 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lvn\" (UniqueName: \"kubernetes.io/projected/2ad2dae2-3cce-4572-94bc-8f67b2837c98-kube-api-access-p2lvn\") pod \"maas-api-revoke-sub-59f7c92e-xghbd\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.595612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.595343 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:38.695712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.695323 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 16:32:38.695712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.695400 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 16:32:38.901627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:38.901596 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6"] Nov 03 16:32:38.902226 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:38.902147 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04b1cde_9c26_4dbb_a090_f3e6120b2f26.slice/crio-9caa3f18ffb54fdb4dfe313836d2b91131422557eb485614729b5c188571a68e WatchSource:0}: Error finding container 9caa3f18ffb54fdb4dfe313836d2b91131422557eb485614729b5c188571a68e: Status 404 returned error can't find the container with id 9caa3f18ffb54fdb4dfe313836d2b91131422557eb485614729b5c188571a68e Nov 03 16:32:39.013294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.013262 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd"] Nov 03 16:32:39.014594 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:39.014561 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad2dae2_3cce_4572_94bc_8f67b2837c98.slice/crio-fa09da958a1c1aa97d1ee1730060febe0d19f7f0264c98a3bf89a14354a69098 WatchSource:0}: Error finding container fa09da958a1c1aa97d1ee1730060febe0d19f7f0264c98a3bf89a14354a69098: Status 404 returned error can't find the container with id fa09da958a1c1aa97d1ee1730060febe0d19f7f0264c98a3bf89a14354a69098 Nov 03 16:32:39.105085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.105046 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" event={"ID":"f04b1cde-9c26-4dbb-a090-f3e6120b2f26","Type":"ContainerStarted","Data":"9caa3f18ffb54fdb4dfe313836d2b91131422557eb485614729b5c188571a68e"} Nov 03 16:32:39.106757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.106724 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" event={"ID":"2ad2dae2-3cce-4572-94bc-8f67b2837c98","Type":"ContainerStarted","Data":"fa09da958a1c1aa97d1ee1730060febe0d19f7f0264c98a3bf89a14354a69098"} Nov 03 16:32:39.604787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.604750 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl"] Nov 03 16:32:39.610653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.610611 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.719140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.719106 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl"] Nov 03 16:32:39.769339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.769301 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.782507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.782477 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fccc840-a6fe-4506-a7f0-da112c251f2c-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.782704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.782686 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tncsl\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-kube-api-access-tncsl\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.782847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.782831 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.782964 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.782949 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.783080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.783066 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.783277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.783258 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.783435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.783419 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.783600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.783582 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.883808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.883732 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:39.884674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884646 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.884810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884685 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fccc840-a6fe-4506-a7f0-da112c251f2c-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.884810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884736 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tncsl\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-kube-api-access-tncsl\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.884810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884773 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.884810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884803 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.885031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884829 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.885031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884933 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.885031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.884977 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.885175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.885047 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.885175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.885146 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.885577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.885536 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:39.885846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.885820 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.886499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.886473 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.886499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.886492 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:32:39.886653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.886520 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fccc840-a6fe-4506-a7f0-da112c251f2c-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.886877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.886850 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.887857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.887838 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.888552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.888533 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.921875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.921839 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:39.922251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.922197 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:39.923066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.923046 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:32:39.945439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.945417 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.945965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.945941 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tncsl\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-kube-api-access-tncsl\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:39.956016 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:39.955993 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:40.116388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.116351 5946 generic.go:356] "Generic (PLEG): container finished" podID="2ad2dae2-3cce-4572-94bc-8f67b2837c98" containerID="6058cdab0483d18514bd88f1e99510f6451f19ec2788fbbd836a776ac120d20a" exitCode=0 Nov 03 16:32:40.116545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.116431 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" event={"ID":"2ad2dae2-3cce-4572-94bc-8f67b2837c98","Type":"ContainerDied","Data":"6058cdab0483d18514bd88f1e99510f6451f19ec2788fbbd836a776ac120d20a"} Nov 03 16:32:40.118176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.118149 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" event={"ID":"f04b1cde-9c26-4dbb-a090-f3e6120b2f26","Type":"ContainerStarted","Data":"188ae47ac10c5af56a66f83b0b5c402fce5c1e3713fa3bcbc48f8978bfdaaec5"} Nov 03 16:32:40.384941 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:40.384904 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fccc840_a6fe_4506_a7f0_da112c251f2c.slice/crio-f57f05fe77c20a80187e4cd603bd4410ee8fe9249c59e8978d3c15a39a0074ac WatchSource:0}: Error finding container f57f05fe77c20a80187e4cd603bd4410ee8fe9249c59e8978d3c15a39a0074ac: Status 404 returned error can't find the container with id f57f05fe77c20a80187e4cd603bd4410ee8fe9249c59e8978d3c15a39a0074ac Nov 03 16:32:40.385500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.385471 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl"] Nov 03 16:32:40.388177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.385916 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" podStartSLOduration=2.385898756 podStartE2EDuration="2.385898756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:40.382871756 +0000 UTC m=+1230.503924581" watchObservedRunningTime="2026-11-03 16:32:40.385898756 +0000 UTC m=+1230.506951579" Nov 03 16:32:40.392815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.392778 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:40.392948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.392853 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:40.392948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.392888 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:40.404642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.404617 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:40.976959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.976925 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp"] Nov 03 16:32:40.981331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:40.981309 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.044965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.044929 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp"] Nov 03 16:32:41.104648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.104604 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.104648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.104649 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5sh\" (UniqueName: \"kubernetes.io/projected/7d4903f4-eaee-4f52-9a86-3622ba5d5427-kube-api-access-cm5sh\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.104910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.104895 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.155191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.155149 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" event={"ID":"7fccc840-a6fe-4506-a7f0-da112c251f2c","Type":"ContainerStarted","Data":"59e041301d52c3192642df3e7ecf64e2e8358326dee9489f58663f41fcec4304"} Nov 03 16:32:41.155377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.155275 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" event={"ID":"7fccc840-a6fe-4506-a7f0-da112c251f2c","Type":"ContainerStarted","Data":"f57f05fe77c20a80187e4cd603bd4410ee8fe9249c59e8978d3c15a39a0074ac"} Nov 03 16:32:41.157055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.157030 5946 generic.go:356] "Generic (PLEG): container finished" podID="f04b1cde-9c26-4dbb-a090-f3e6120b2f26" containerID="188ae47ac10c5af56a66f83b0b5c402fce5c1e3713fa3bcbc48f8978bfdaaec5" exitCode=0 Nov 03 16:32:41.157177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.157163 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" event={"ID":"f04b1cde-9c26-4dbb-a090-f3e6120b2f26","Type":"ContainerDied","Data":"188ae47ac10c5af56a66f83b0b5c402fce5c1e3713fa3bcbc48f8978bfdaaec5"} Nov 03 16:32:41.206183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.206142 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.206384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.206365 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.206435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.206398 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5sh\" (UniqueName: \"kubernetes.io/projected/7d4903f4-eaee-4f52-9a86-3622ba5d5427-kube-api-access-cm5sh\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.206923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.206901 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.208812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.208788 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.269693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.269658 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5sh\" (UniqueName: \"kubernetes.io/projected/7d4903f4-eaee-4f52-9a86-3622ba5d5427-kube-api-access-cm5sh\") pod \"maas-api-revoke-sub-3dba9775-4h7jp\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.284637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.284573 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podStartSLOduration=2.284552122 podStartE2EDuration="2.284552122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:41.280050585 +0000 UTC m=+1231.401103412" watchObservedRunningTime="2026-11-03 16:32:41.284552122 +0000 UTC m=+1231.405604947" Nov 03 16:32:41.295570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.295541 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:41.366572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.366544 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:41.512435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.512351 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-xks-ca\") pod \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " Nov 03 16:32:41.512435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.512418 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-openshift-ca\") pod \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " Nov 03 16:32:41.512685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.512630 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad2dae2-3cce-4572-94bc-8f67b2837c98-kube-api-access-p2lvn\" (UniqueName: \"kubernetes.io/projected/2ad2dae2-3cce-4572-94bc-8f67b2837c98-kube-api-access-p2lvn\") pod \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\" (UID: \"2ad2dae2-3cce-4572-94bc-8f67b2837c98\") " Nov 03 16:32:41.512836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.512803 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-openshift-ca" pod "2ad2dae2-3cce-4572-94bc-8f67b2837c98" (UID: "2ad2dae2-3cce-4572-94bc-8f67b2837c98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:41.513133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.513108 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:41.541841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.541778 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad2dae2-3cce-4572-94bc-8f67b2837c98-kube-api-access-p2lvn" pod "2ad2dae2-3cce-4572-94bc-8f67b2837c98" (UID: "2ad2dae2-3cce-4572-94bc-8f67b2837c98"). InnerVolumeSpecName "kube-api-access-p2lvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:41.544940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.544851 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-xks-ca" pod "2ad2dae2-3cce-4572-94bc-8f67b2837c98" (UID: "2ad2dae2-3cce-4572-94bc-8f67b2837c98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:41.639085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.634341 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lvn\" (UniqueName: \"kubernetes.io/projected/2ad2dae2-3cce-4572-94bc-8f67b2837c98-kube-api-access-p2lvn\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:41.639085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.634394 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2dae2-3cce-4572-94bc-8f67b2837c98-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:41.712458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.712426 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp"] Nov 03 16:32:41.712713 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:41.712691 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4903f4_eaee_4f52_9a86_3622ba5d5427.slice/crio-0e6322f9a7bd9d2808dcb7908e98be9e1ffbfb01a7ce073c05975a193c3264f5 WatchSource:0}: Error finding container 0e6322f9a7bd9d2808dcb7908e98be9e1ffbfb01a7ce073c05975a193c3264f5: Status 404 returned error can't find the container with id 0e6322f9a7bd9d2808dcb7908e98be9e1ffbfb01a7ce073c05975a193c3264f5 Nov 03 16:32:41.956280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.956245 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:41.958110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.958086 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:32:41.958258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:41.958136 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 16:32:42.167003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.166978 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" Nov 03 16:32:42.167456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.167013 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd" event={"ID":"2ad2dae2-3cce-4572-94bc-8f67b2837c98","Type":"ContainerDied","Data":"fa09da958a1c1aa97d1ee1730060febe0d19f7f0264c98a3bf89a14354a69098"} Nov 03 16:32:42.167456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.167064 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa09da958a1c1aa97d1ee1730060febe0d19f7f0264c98a3bf89a14354a69098" Nov 03 16:32:42.168970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.168942 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" event={"ID":"7d4903f4-eaee-4f52-9a86-3622ba5d5427","Type":"ContainerStarted","Data":"412659bf81e5a849139342c6dc50d0e2bc1c05cda4a2410df5ecee4e0f9578c7"} Nov 03 16:32:42.169121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.168994 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" event={"ID":"7d4903f4-eaee-4f52-9a86-3622ba5d5427","Type":"ContainerStarted","Data":"0e6322f9a7bd9d2808dcb7908e98be9e1ffbfb01a7ce073c05975a193c3264f5"} Nov 03 16:32:42.301379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.301327 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" podStartSLOduration=2.3013126489999998 podStartE2EDuration="2.301312649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:42.299403006 +0000 UTC m=+1232.420455833" watchObservedRunningTime="2026-11-03 16:32:42.301312649 +0000 UTC m=+1232.422365472" Nov 03 16:32:42.338371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.330413 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:42.452726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.452697 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-xks-ca\") pod \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " Nov 03 16:32:42.452914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.452896 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-openshift-ca\") pod \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " Nov 03 16:32:42.453037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.453023 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-kube-api-access-g6qcw\" (UniqueName: \"kubernetes.io/projected/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-kube-api-access-g6qcw\") pod \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\" (UID: \"f04b1cde-9c26-4dbb-a090-f3e6120b2f26\") " Nov 03 16:32:42.453250 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.453188 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-openshift-ca" pod "f04b1cde-9c26-4dbb-a090-f3e6120b2f26" (UID: "f04b1cde-9c26-4dbb-a090-f3e6120b2f26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:42.453534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.453517 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:42.454898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.454871 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-xks-ca" pod "f04b1cde-9c26-4dbb-a090-f3e6120b2f26" (UID: "f04b1cde-9c26-4dbb-a090-f3e6120b2f26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:42.455357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.455333 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-kube-api-access-g6qcw" pod "f04b1cde-9c26-4dbb-a090-f3e6120b2f26" (UID: "f04b1cde-9c26-4dbb-a090-f3e6120b2f26"). InnerVolumeSpecName "kube-api-access-g6qcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:42.554963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.554931 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6qcw\" (UniqueName: \"kubernetes.io/projected/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-kube-api-access-g6qcw\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:42.554963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.554963 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04b1cde-9c26-4dbb-a090-f3e6120b2f26-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:42.586566 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:32:42.586528 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:32:42.957097 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.957064 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:32:42.957287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:42.957117 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 16:32:43.061480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.061438 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw"] Nov 03 16:32:43.066063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.066031 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdd9168-m55zw"] Nov 03 16:32:43.180071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.179996 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" Nov 03 16:32:43.180513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.179996 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6" event={"ID":"f04b1cde-9c26-4dbb-a090-f3e6120b2f26","Type":"ContainerDied","Data":"9caa3f18ffb54fdb4dfe313836d2b91131422557eb485614729b5c188571a68e"} Nov 03 16:32:43.180513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.180133 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9caa3f18ffb54fdb4dfe313836d2b91131422557eb485614729b5c188571a68e" Nov 03 16:32:43.181929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.181902 5946 generic.go:356] "Generic (PLEG): container finished" podID="7d4903f4-eaee-4f52-9a86-3622ba5d5427" containerID="412659bf81e5a849139342c6dc50d0e2bc1c05cda4a2410df5ecee4e0f9578c7" exitCode=0 Nov 03 16:32:43.182036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.181986 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" event={"ID":"7d4903f4-eaee-4f52-9a86-3622ba5d5427","Type":"ContainerDied","Data":"412659bf81e5a849139342c6dc50d0e2bc1c05cda4a2410df5ecee4e0f9578c7"} Nov 03 16:32:43.957360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.957322 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:32:43.957520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:43.957381 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 16:32:44.004599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.004568 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62825154-7181-400a-bae3-5f893371bb43" path="/var/lib/kubelet/pods/62825154-7181-400a-bae3-5f893371bb43/volumes" Nov 03 16:32:44.382558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.382536 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:44.478752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.478670 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-openshift-ca\") pod \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " Nov 03 16:32:44.478903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.478769 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-xks-ca\") pod \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " Nov 03 16:32:44.478903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.478791 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4903f4-eaee-4f52-9a86-3622ba5d5427-kube-api-access-cm5sh\" (UniqueName: \"kubernetes.io/projected/7d4903f4-eaee-4f52-9a86-3622ba5d5427-kube-api-access-cm5sh\") pod \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\" (UID: \"7d4903f4-eaee-4f52-9a86-3622ba5d5427\") " Nov 03 16:32:44.479103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.479074 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-openshift-ca" pod "7d4903f4-eaee-4f52-9a86-3622ba5d5427" (UID: "7d4903f4-eaee-4f52-9a86-3622ba5d5427"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:44.479273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.479256 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:44.481065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.481039 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-xks-ca" pod "7d4903f4-eaee-4f52-9a86-3622ba5d5427" (UID: "7d4903f4-eaee-4f52-9a86-3622ba5d5427"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:44.481065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.481053 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4903f4-eaee-4f52-9a86-3622ba5d5427-kube-api-access-cm5sh" pod "7d4903f4-eaee-4f52-9a86-3622ba5d5427" (UID: "7d4903f4-eaee-4f52-9a86-3622ba5d5427"). InnerVolumeSpecName "kube-api-access-cm5sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:44.580688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.580656 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4903f4-eaee-4f52-9a86-3622ba5d5427-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:44.580688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.580687 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm5sh\" (UniqueName: \"kubernetes.io/projected/7d4903f4-eaee-4f52-9a86-3622ba5d5427-kube-api-access-cm5sh\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:44.956838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.956807 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:32:44.957021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:44.956858 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 16:32:45.200063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.200035 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" Nov 03 16:32:45.200245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.200034 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp" event={"ID":"7d4903f4-eaee-4f52-9a86-3622ba5d5427","Type":"ContainerDied","Data":"0e6322f9a7bd9d2808dcb7908e98be9e1ffbfb01a7ce073c05975a193c3264f5"} Nov 03 16:32:45.200245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.200161 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6322f9a7bd9d2808dcb7908e98be9e1ffbfb01a7ce073c05975a193c3264f5" Nov 03 16:32:45.208889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.208784 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:45.543922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.543883 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4"] Nov 03 16:32:45.545442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545404 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2ad2dae2-3cce-4572-94bc-8f67b2837c98" containerName="revoke-keys" Nov 03 16:32:45.545442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545430 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad2dae2-3cce-4572-94bc-8f67b2837c98" containerName="revoke-keys" Nov 03 16:32:45.545628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545449 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7d4903f4-eaee-4f52-9a86-3622ba5d5427" containerName="revoke-keys" Nov 03 16:32:45.545628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545457 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4903f4-eaee-4f52-9a86-3622ba5d5427" containerName="revoke-keys" Nov 03 16:32:45.545628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545512 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f04b1cde-9c26-4dbb-a090-f3e6120b2f26" containerName="revoke-keys" Nov 03 16:32:45.545628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545520 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04b1cde-9c26-4dbb-a090-f3e6120b2f26" containerName="revoke-keys" Nov 03 16:32:45.545628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545630 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04b1cde-9c26-4dbb-a090-f3e6120b2f26" containerName="revoke-keys" Nov 03 16:32:45.545860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545648 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad2dae2-3cce-4572-94bc-8f67b2837c98" containerName="revoke-keys" Nov 03 16:32:45.545860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.545657 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4903f4-eaee-4f52-9a86-3622ba5d5427" containerName="revoke-keys" Nov 03 16:32:45.549447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.549430 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.593769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.593736 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.593921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.593904 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.594036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.594011 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b764326c-bef1-4e49-bb24-e53f23b38526-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.594094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.594076 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.594161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.594143 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.594198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.594170 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.594342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.594322 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.594389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.594366 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b764326c-bef1-4e49-bb24-e53f23b38526-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.594428 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.594391 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwk9\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-kube-api-access-lpwk9\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.643442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.643402 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4"] Nov 03 16:32:45.696113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696073 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696110 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696183 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696237 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b764326c-bef1-4e49-bb24-e53f23b38526-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696261 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwk9\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-kube-api-access-lpwk9\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696345 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696496 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696510 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696547 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b764326c-bef1-4e49-bb24-e53f23b38526-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696601 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696718 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.696962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696938 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b764326c-bef1-4e49-bb24-e53f23b38526-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.697045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696994 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.697045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.696996 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.698619 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.698595 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.699056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.699038 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b764326c-bef1-4e49-bb24-e53f23b38526-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.759880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.759843 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwk9\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-kube-api-access-lpwk9\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.773526 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.773499 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.864427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.864346 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:45.957997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.957520 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:32:45.957997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:45.957589 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 16:32:46.200990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.200906 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt"] Nov 03 16:32:46.206131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.206109 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.269131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.269096 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt"] Nov 03 16:32:46.303736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.303703 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.303929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.303749 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.303929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.303883 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.303929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.303922 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.304096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.303948 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.304096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.304000 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjp5j\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-kube-api-access-zjp5j\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.304096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.304052 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.304096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.304084 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.304311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.304146 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.370171 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:46.370139 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb764326c_bef1_4e49_bb24_e53f23b38526.slice/crio-6e4df9c9c9c560e8016878bec25c57b4d4d5dde0fe7bc862e677ad40d1838920 WatchSource:0}: Error finding container 6e4df9c9c9c560e8016878bec25c57b4d4d5dde0fe7bc862e677ad40d1838920: Status 404 returned error can't find the container with id 6e4df9c9c9c560e8016878bec25c57b4d4d5dde0fe7bc862e677ad40d1838920 Nov 03 16:32:46.370298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.370188 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4"] Nov 03 16:32:46.372943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.372912 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:46.373056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.372974 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:46.373056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.373005 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:46.405656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405456 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405529 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405562 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405619 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjp5j\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-kube-api-access-zjp5j\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405677 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405706 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405770 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405871 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.405977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.405931 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.406527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.406336 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.406637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.406553 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.406715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.406652 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.406780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.406704 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.407094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.407074 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.408498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.408481 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.408809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.408794 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.465878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.465847 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjp5j\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-kube-api-access-zjp5j\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.465982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.465873 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.547575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.547538 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:46.963721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.963679 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:32:46.963890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:46.963750 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 16:32:47.004299 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:47.004258 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037cce48_42bf_4fe3_9a0a_a5fc6691047f.slice/crio-57a0092051efe6ac6045b46d5d78cb7d5e98812e2aadeb4a728a34cb77621af0 WatchSource:0}: Error finding container 57a0092051efe6ac6045b46d5d78cb7d5e98812e2aadeb4a728a34cb77621af0: Status 404 returned error can't find the container with id 57a0092051efe6ac6045b46d5d78cb7d5e98812e2aadeb4a728a34cb77621af0 Nov 03 16:32:47.006337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.006308 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt"] Nov 03 16:32:47.007140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.007101 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:47.007221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.007191 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:47.007267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.007252 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:32:47.252459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.252349 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" event={"ID":"b764326c-bef1-4e49-bb24-e53f23b38526","Type":"ContainerStarted","Data":"84dd1ac35b0c3dd40e5ed9e0c4e3a16a580e3ceea5c0847054a4e30a6ea55de8"} Nov 03 16:32:47.252459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.252419 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" event={"ID":"b764326c-bef1-4e49-bb24-e53f23b38526","Type":"ContainerStarted","Data":"6e4df9c9c9c560e8016878bec25c57b4d4d5dde0fe7bc862e677ad40d1838920"} Nov 03 16:32:47.255970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.255193 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" event={"ID":"037cce48-42bf-4fe3-9a0a-a5fc6691047f","Type":"ContainerStarted","Data":"f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79"} Nov 03 16:32:47.255970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.255261 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" event={"ID":"037cce48-42bf-4fe3-9a0a-a5fc6691047f","Type":"ContainerStarted","Data":"57a0092051efe6ac6045b46d5d78cb7d5e98812e2aadeb4a728a34cb77621af0"} Nov 03 16:32:47.384230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.383764 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podStartSLOduration=2.383745752 podStartE2EDuration="2.383745752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:47.377375662 +0000 UTC m=+1237.498428495" watchObservedRunningTime="2026-11-03 16:32:47.383745752 +0000 UTC m=+1237.504798583" Nov 03 16:32:47.511675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.511550 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podStartSLOduration=1.5115304429999998 podStartE2EDuration="1.511530443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:47.505808359 +0000 UTC m=+1237.626861222" watchObservedRunningTime="2026-11-03 16:32:47.511530443 +0000 UTC m=+1237.632583264" Nov 03 16:32:47.544102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.544062 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc"] Nov 03 16:32:47.565612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.565543 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:47.566290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.566200 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.625233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.624620 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:32:47.625448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.625394 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:32:47.625542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.625501 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:32:47.625589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.625557 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:32:47.682231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.681187 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.682231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.681317 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.682231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.681375 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8h8\" (UniqueName: \"kubernetes.io/projected/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-kube-api-access-bk8h8\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.682231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.682078 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc"] Nov 03 16:32:47.789144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.789057 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.789898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.789873 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.790068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.790053 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8h8\" (UniqueName: \"kubernetes.io/projected/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-kube-api-access-bk8h8\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.791832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.791303 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.864260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.862663 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:47.879243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.868553 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:47.904044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.903670 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:32:47.904044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.903741 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:32:47.904386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:47.903537 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8h8\" (UniqueName: \"kubernetes.io/projected/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-kube-api-access-bk8h8\") pod \"maas-api-revoke-sub-99640711-vd2zc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:48.169285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.168792 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:48.171631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.170636 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:48.171893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.171870 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:32:48.184001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.183982 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:48.572433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.572360 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:32:48.573031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.572984 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 16:32:48.670753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.670704 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc"] Nov 03 16:32:48.673947 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:48.673917 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfbdc5c_243d_4a5e_a343_1d7e39c842fc.slice/crio-a82f57bb05e5dde1e41ee88f010d2b13351be9be3d75819b0df4639a833cf7dc WatchSource:0}: Error finding container a82f57bb05e5dde1e41ee88f010d2b13351be9be3d75819b0df4639a833cf7dc: Status 404 returned error can't find the container with id a82f57bb05e5dde1e41ee88f010d2b13351be9be3d75819b0df4639a833cf7dc Nov 03 16:32:48.866923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.866656 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:32:48.866923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:48.866721 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:32:49.166316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.162274 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz"] Nov 03 16:32:49.169173 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.169143 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.194435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.194274 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz"] Nov 03 16:32:49.276314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.276268 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.276688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.276663 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthrz\" (UniqueName: \"kubernetes.io/projected/40bae6f3-5757-48ac-92f5-441867cb0c28-kube-api-access-nthrz\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.276897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.276880 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.294808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.294781 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" event={"ID":"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc","Type":"ContainerStarted","Data":"2f04c33b03d0d9ece2c0cba008b3ea910abb47add5f97755fe6a0ef606ed38fb"} Nov 03 16:32:49.295017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.294988 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" event={"ID":"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc","Type":"ContainerStarted","Data":"a82f57bb05e5dde1e41ee88f010d2b13351be9be3d75819b0df4639a833cf7dc"} Nov 03 16:32:49.378711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.378674 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthrz\" (UniqueName: \"kubernetes.io/projected/40bae6f3-5757-48ac-92f5-441867cb0c28-kube-api-access-nthrz\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.378900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.378753 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.378900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.378839 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.379504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.379486 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.382011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.381992 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.473319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.465389 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" podStartSLOduration=2.465368176 podStartE2EDuration="2.465368176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:49.429654262 +0000 UTC m=+1239.550707096" watchObservedRunningTime="2026-11-03 16:32:49.465368176 +0000 UTC m=+1239.586421000" Nov 03 16:32:49.474465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.468289 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthrz\" (UniqueName: \"kubernetes.io/projected/40bae6f3-5757-48ac-92f5-441867cb0c28-kube-api-access-nthrz\") pod \"maas-api-revoke-sub-6c7ae4bc-njhrz\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.484713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.484338 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:49.561234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.550655 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:32:49.561234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.550717 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 16:32:49.867601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.866990 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:32:49.867601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.867053 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:32:49.957526 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:49.955642 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz"] Nov 03 16:32:50.275263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.275159 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:50.306787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.306744 5946 generic.go:356] "Generic (PLEG): container finished" podID="1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" containerID="2f04c33b03d0d9ece2c0cba008b3ea910abb47add5f97755fe6a0ef606ed38fb" exitCode=0 Nov 03 16:32:50.306962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.306825 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" event={"ID":"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc","Type":"ContainerDied","Data":"2f04c33b03d0d9ece2c0cba008b3ea910abb47add5f97755fe6a0ef606ed38fb"} Nov 03 16:32:50.308352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.308325 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" event={"ID":"40bae6f3-5757-48ac-92f5-441867cb0c28","Type":"ContainerStarted","Data":"9de128695d8c85e361cc08669fa90be4597a0baee1c2dee3380ab0987efc1db7"} Nov 03 16:32:50.567710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.567404 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:32:50.567710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.567466 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 16:32:50.866289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.865509 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:32:50.866289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:50.865570 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:32:51.370235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.367232 5946 generic.go:356] "Generic (PLEG): container finished" podID="40bae6f3-5757-48ac-92f5-441867cb0c28" containerID="21e19d64a9f36e40d981c95dbeb34b988f19df9ff704e40ed0265eddbc526a96" exitCode=0 Nov 03 16:32:51.370235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.367935 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" event={"ID":"40bae6f3-5757-48ac-92f5-441867cb0c28","Type":"ContainerDied","Data":"21e19d64a9f36e40d981c95dbeb34b988f19df9ff704e40ed0265eddbc526a96"} Nov 03 16:32:51.549750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.549641 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:32:51.549750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.549708 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 16:32:51.621256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.612723 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:51.738232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.733808 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-xks-ca\") pod \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " Nov 03 16:32:51.738232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.733923 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-kube-api-access-bk8h8\" (UniqueName: \"kubernetes.io/projected/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-kube-api-access-bk8h8\") pod \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " Nov 03 16:32:51.738232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.734068 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-openshift-ca\") pod \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\" (UID: \"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc\") " Nov 03 16:32:51.738232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.735075 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-openshift-ca" pod "1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" (UID: "1dfbdc5c-243d-4a5e-a343-1d7e39c842fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:51.742228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.741377 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-kube-api-access-bk8h8" pod "1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" (UID: "1dfbdc5c-243d-4a5e-a343-1d7e39c842fc"). InnerVolumeSpecName "kube-api-access-bk8h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:51.772231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.768540 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-xks-ca" pod "1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" (UID: "1dfbdc5c-243d-4a5e-a343-1d7e39c842fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:51.843289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.842897 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:51.843289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.842935 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk8h8\" (UniqueName: \"kubernetes.io/projected/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-kube-api-access-bk8h8\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:51.843289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.842952 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:51.868177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.867692 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:32:51.868177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:51.867758 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:32:52.086031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.086005 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5"] Nov 03 16:32:52.090320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.090303 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c48fae-tllm5"] Nov 03 16:32:52.379886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.379858 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" Nov 03 16:32:52.380389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.380355 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc" event={"ID":"1dfbdc5c-243d-4a5e-a343-1d7e39c842fc","Type":"ContainerDied","Data":"a82f57bb05e5dde1e41ee88f010d2b13351be9be3d75819b0df4639a833cf7dc"} Nov 03 16:32:52.380467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.380433 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a82f57bb05e5dde1e41ee88f010d2b13351be9be3d75819b0df4639a833cf7dc" Nov 03 16:32:52.557388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.557348 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:32:52.557598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.557408 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 16:32:52.647893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.647802 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k"] Nov 03 16:32:52.648832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.648801 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" containerName="revoke-keys" Nov 03 16:32:52.648832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.648831 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" containerName="revoke-keys" Nov 03 16:32:52.649053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.649036 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" containerName="revoke-keys" Nov 03 16:32:52.653760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.653738 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.658178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.658159 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:52.677361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.677331 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k"] Nov 03 16:32:52.789616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.789585 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bae6f3-5757-48ac-92f5-441867cb0c28-kube-api-access-nthrz\" (UniqueName: \"kubernetes.io/projected/40bae6f3-5757-48ac-92f5-441867cb0c28-kube-api-access-nthrz\") pod \"40bae6f3-5757-48ac-92f5-441867cb0c28\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " Nov 03 16:32:52.789841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.789749 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-openshift-ca\") pod \"40bae6f3-5757-48ac-92f5-441867cb0c28\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " Nov 03 16:32:52.789841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.789789 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-xks-ca\") pod \"40bae6f3-5757-48ac-92f5-441867cb0c28\" (UID: \"40bae6f3-5757-48ac-92f5-441867cb0c28\") " Nov 03 16:32:52.790371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.790340 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-openshift-ca" pod "40bae6f3-5757-48ac-92f5-441867cb0c28" (UID: "40bae6f3-5757-48ac-92f5-441867cb0c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:52.790551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.790385 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.790663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.790606 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48sb\" (UniqueName: \"kubernetes.io/projected/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-kube-api-access-g48sb\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.790769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.790662 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.790925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.790892 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:52.792600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.792573 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bae6f3-5757-48ac-92f5-441867cb0c28-kube-api-access-nthrz" pod "40bae6f3-5757-48ac-92f5-441867cb0c28" (UID: "40bae6f3-5757-48ac-92f5-441867cb0c28"). InnerVolumeSpecName "kube-api-access-nthrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:52.792912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.792884 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-xks-ca" pod "40bae6f3-5757-48ac-92f5-441867cb0c28" (UID: "40bae6f3-5757-48ac-92f5-441867cb0c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:52.865483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.865449 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:32:52.865681 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.865520 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:32:52.892088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.892052 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.892290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.892115 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48sb\" (UniqueName: \"kubernetes.io/projected/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-kube-api-access-g48sb\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.892290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.892156 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.892290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.892282 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bae6f3-5757-48ac-92f5-441867cb0c28-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:52.892454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.892299 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nthrz\" (UniqueName: \"kubernetes.io/projected/40bae6f3-5757-48ac-92f5-441867cb0c28-kube-api-access-nthrz\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:52.892895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.892875 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.895526 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.895376 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.953109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.953040 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48sb\" (UniqueName: \"kubernetes.io/projected/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-kube-api-access-g48sb\") pod \"maas-api-revoke-sub-5a459b49-2cd9k\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:52.969921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:52.969890 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:53.387296 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:53.387258 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94df55e4_bbc0_4c88_8ece_92ce1d6f0d58.slice/crio-4642d6ff2149bcb65738e5951c8dd917fddfd5b93f66a597dd309b4d13d707eb WatchSource:0}: Error finding container 4642d6ff2149bcb65738e5951c8dd917fddfd5b93f66a597dd309b4d13d707eb: Status 404 returned error can't find the container with id 4642d6ff2149bcb65738e5951c8dd917fddfd5b93f66a597dd309b4d13d707eb Nov 03 16:32:53.388348 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:53.388317 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k"] Nov 03 16:32:53.392521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:53.392492 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" event={"ID":"40bae6f3-5757-48ac-92f5-441867cb0c28","Type":"ContainerDied","Data":"9de128695d8c85e361cc08669fa90be4597a0baee1c2dee3380ab0987efc1db7"} Nov 03 16:32:53.392628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:53.392540 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de128695d8c85e361cc08669fa90be4597a0baee1c2dee3380ab0987efc1db7" Nov 03 16:32:53.392628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:53.392558 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz" Nov 03 16:32:53.548273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:53.548237 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:32:53.548464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:53.548294 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 16:32:54.008941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.008891 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78f6eb7-6c6a-4fd9-9597-6b56b3361058" path="/var/lib/kubelet/pods/c78f6eb7-6c6a-4fd9-9597-6b56b3361058/volumes" Nov 03 16:32:54.029619 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.029591 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:54.031102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.031022 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:54.032430 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.032406 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:32:54.412007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.411918 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" event={"ID":"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58","Type":"ContainerStarted","Data":"8155552fccb170afce2b8d9007d6dfc2b8b2ccac70eb8f5b2cf7646c25c385ef"} Nov 03 16:32:54.412007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.411975 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" event={"ID":"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58","Type":"ContainerStarted","Data":"4642d6ff2149bcb65738e5951c8dd917fddfd5b93f66a597dd309b4d13d707eb"} Nov 03 16:32:54.541304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.541193 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" podStartSLOduration=2.541173426 podStartE2EDuration="2.541173426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:54.540895599 +0000 UTC m=+1244.661948432" watchObservedRunningTime="2026-11-03 16:32:54.541173426 +0000 UTC m=+1244.662226249" Nov 03 16:32:54.666689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.666587 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:54.667551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.667441 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:54.668290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:54.668264 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:32:55.197800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:55.197767 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:55.422747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:55.422716 5946 generic.go:356] "Generic (PLEG): container finished" podID="94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" containerID="8155552fccb170afce2b8d9007d6dfc2b8b2ccac70eb8f5b2cf7646c25c385ef" exitCode=0 Nov 03 16:32:55.423154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:55.422803 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" event={"ID":"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58","Type":"ContainerDied","Data":"8155552fccb170afce2b8d9007d6dfc2b8b2ccac70eb8f5b2cf7646c25c385ef"} Nov 03 16:32:56.603566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.603540 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:56.747190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.746816 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-xks-ca\") pod \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " Nov 03 16:32:56.747190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.746913 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-kube-api-access-g48sb\" (UniqueName: \"kubernetes.io/projected/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-kube-api-access-g48sb\") pod \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " Nov 03 16:32:56.747190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.747021 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-openshift-ca\") pod \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\" (UID: \"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58\") " Nov 03 16:32:56.747728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.747696 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-openshift-ca" pod "94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" (UID: "94df55e4-bbc0-4c88-8ece-92ce1d6f0d58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:32:56.747845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.747833 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:56.749992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.749965 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-kube-api-access-g48sb" pod "94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" (UID: "94df55e4-bbc0-4c88-8ece-92ce1d6f0d58"). InnerVolumeSpecName "kube-api-access-g48sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:56.750096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.750050 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-xks-ca" pod "94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" (UID: "94df55e4-bbc0-4c88-8ece-92ce1d6f0d58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:56.849323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.849282 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:56.849323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:56.849320 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48sb\" (UniqueName: \"kubernetes.io/projected/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58-kube-api-access-g48sb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:32:57.442452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:57.442418 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" Nov 03 16:32:57.442652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:57.442448 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k" event={"ID":"94df55e4-bbc0-4c88-8ece-92ce1d6f0d58","Type":"ContainerDied","Data":"4642d6ff2149bcb65738e5951c8dd917fddfd5b93f66a597dd309b4d13d707eb"} Nov 03 16:32:57.442652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:57.442512 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4642d6ff2149bcb65738e5951c8dd917fddfd5b93f66a597dd309b4d13d707eb" Nov 03 16:32:58.229016 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.228985 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm"] Nov 03 16:32:58.229621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.229604 5946 cpu_manager.go:414] "containerMap: removing container" podUID="40bae6f3-5757-48ac-92f5-441867cb0c28" containerName="revoke-keys" Nov 03 16:32:58.229661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.229622 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bae6f3-5757-48ac-92f5-441867cb0c28" containerName="revoke-keys" Nov 03 16:32:58.229661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.229649 5946 cpu_manager.go:414] "containerMap: removing container" podUID="94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" containerName="revoke-keys" Nov 03 16:32:58.229661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.229655 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" containerName="revoke-keys" Nov 03 16:32:58.229767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.229762 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" containerName="revoke-keys" Nov 03 16:32:58.229797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.229778 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bae6f3-5757-48ac-92f5-441867cb0c28" containerName="revoke-keys" Nov 03 16:32:58.233571 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.233551 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.293759 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.293734 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm"] Nov 03 16:32:58.339875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.339840 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:32:58.339875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.339850 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:32:58.339875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.339850 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:32:58.340078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.339903 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:32:58.368369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.368338 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqfj\" (UniqueName: \"kubernetes.io/projected/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-kube-api-access-jfqfj\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.368525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.368400 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.368563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.368517 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.469934 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.469904 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqfj\" (UniqueName: \"kubernetes.io/projected/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-kube-api-access-jfqfj\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.470094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.469966 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.470094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.470031 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.470654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.470632 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.472598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.472578 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.531565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.531537 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqfj\" (UniqueName: \"kubernetes.io/projected/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-kube-api-access-jfqfj\") pod \"maas-api-revoke-sub-78f5709b-wpclm\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.548081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.548058 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:32:58.936182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:58.936149 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm"] Nov 03 16:32:58.936407 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:32:58.936384 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00db8266_7b88_4c8e_920c_0ba95c9b5ccc.slice/crio-061f9c6526c6477da9b774ab4a961959a9d11bcfe90f6bb1157dad9df0e19727 WatchSource:0}: Error finding container 061f9c6526c6477da9b774ab4a961959a9d11bcfe90f6bb1157dad9df0e19727: Status 404 returned error can't find the container with id 061f9c6526c6477da9b774ab4a961959a9d11bcfe90f6bb1157dad9df0e19727 Nov 03 16:32:59.459731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.459689 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" event={"ID":"00db8266-7b88-4c8e-920c-0ba95c9b5ccc","Type":"ContainerStarted","Data":"082005ff86bd8e48af417ad87e360479176b01e592ecdbac71bfb3d4d23b2167"} Nov 03 16:32:59.460161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.459757 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" event={"ID":"00db8266-7b88-4c8e-920c-0ba95c9b5ccc","Type":"ContainerStarted","Data":"061f9c6526c6477da9b774ab4a961959a9d11bcfe90f6bb1157dad9df0e19727"} Nov 03 16:32:59.569285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.569250 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5"] Nov 03 16:32:59.573735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.573686 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.599433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.599389 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" podStartSLOduration=1.599374039 podStartE2EDuration="1.599374039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:59.58099308 +0000 UTC m=+1249.702045904" watchObservedRunningTime="2026-11-03 16:32:59.599374039 +0000 UTC m=+1249.720426861" Nov 03 16:32:59.620397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.620358 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5"] Nov 03 16:32:59.687119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.687080 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.687119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.687132 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.687400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.687313 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m827j\" (UniqueName: \"kubernetes.io/projected/409b02a4-2bb7-4514-b88f-6ae3356071b7-kube-api-access-m827j\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.788533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.788496 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.788533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.788535 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.788739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.788606 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m827j\" (UniqueName: \"kubernetes.io/projected/409b02a4-2bb7-4514-b88f-6ae3356071b7-kube-api-access-m827j\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.789367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.789301 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.791180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.791157 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.849194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.849139 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m827j\" (UniqueName: \"kubernetes.io/projected/409b02a4-2bb7-4514-b88f-6ae3356071b7-kube-api-access-m827j\") pod \"maas-api-revoke-sub-60338208-b8zr5\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.889109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.889086 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:32:59.982241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:32:59.982177 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:00.274091 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:00.274059 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409b02a4_2bb7_4514_b88f_6ae3356071b7.slice/crio-e18684f467b25e3de8bc684e248fb7a3fe8b3ceebac9e23ba589491d8a9379d3 WatchSource:0}: Error finding container e18684f467b25e3de8bc684e248fb7a3fe8b3ceebac9e23ba589491d8a9379d3: Status 404 returned error can't find the container with id e18684f467b25e3de8bc684e248fb7a3fe8b3ceebac9e23ba589491d8a9379d3 Nov 03 16:33:00.274091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:00.274069 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5"] Nov 03 16:33:00.470037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:00.469999 5946 generic.go:356] "Generic (PLEG): container finished" podID="00db8266-7b88-4c8e-920c-0ba95c9b5ccc" containerID="082005ff86bd8e48af417ad87e360479176b01e592ecdbac71bfb3d4d23b2167" exitCode=0 Nov 03 16:33:00.470487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:00.470080 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" event={"ID":"00db8266-7b88-4c8e-920c-0ba95c9b5ccc","Type":"ContainerDied","Data":"082005ff86bd8e48af417ad87e360479176b01e592ecdbac71bfb3d4d23b2167"} Nov 03 16:33:00.471390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:00.471367 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" event={"ID":"409b02a4-2bb7-4514-b88f-6ae3356071b7","Type":"ContainerStarted","Data":"e18684f467b25e3de8bc684e248fb7a3fe8b3ceebac9e23ba589491d8a9379d3"} Nov 03 16:33:01.481854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.481820 5946 generic.go:356] "Generic (PLEG): container finished" podID="409b02a4-2bb7-4514-b88f-6ae3356071b7" containerID="6aa9ada5017d8a9bfc106e4c58b27145ee2307ad799f10682a8f17d1ea49d71b" exitCode=0 Nov 03 16:33:01.482316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.481901 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" event={"ID":"409b02a4-2bb7-4514-b88f-6ae3356071b7","Type":"ContainerDied","Data":"6aa9ada5017d8a9bfc106e4c58b27145ee2307ad799f10682a8f17d1ea49d71b"} Nov 03 16:33:01.654079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.654047 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:33:01.818482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.818400 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-xks-ca\") pod \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " Nov 03 16:33:01.818652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.818533 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-openshift-ca\") pod \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " Nov 03 16:33:01.818652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.818562 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-kube-api-access-jfqfj\" (UniqueName: \"kubernetes.io/projected/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-kube-api-access-jfqfj\") pod \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\" (UID: \"00db8266-7b88-4c8e-920c-0ba95c9b5ccc\") " Nov 03 16:33:01.818945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.818915 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-openshift-ca" pod "00db8266-7b88-4c8e-920c-0ba95c9b5ccc" (UID: "00db8266-7b88-4c8e-920c-0ba95c9b5ccc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:01.819226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.819181 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:01.820757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.820731 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-xks-ca" pod "00db8266-7b88-4c8e-920c-0ba95c9b5ccc" (UID: "00db8266-7b88-4c8e-920c-0ba95c9b5ccc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:01.821354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.821327 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-kube-api-access-jfqfj" pod "00db8266-7b88-4c8e-920c-0ba95c9b5ccc" (UID: "00db8266-7b88-4c8e-920c-0ba95c9b5ccc"). InnerVolumeSpecName "kube-api-access-jfqfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:01.920702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.920667 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfqfj\" (UniqueName: \"kubernetes.io/projected/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-kube-api-access-jfqfj\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:01.920702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:01.920701 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00db8266-7b88-4c8e-920c-0ba95c9b5ccc-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:02.022597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.022566 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq"] Nov 03 16:33:02.023190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.023170 5946 cpu_manager.go:414] "containerMap: removing container" podUID="00db8266-7b88-4c8e-920c-0ba95c9b5ccc" containerName="revoke-keys" Nov 03 16:33:02.023305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.023192 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00db8266-7b88-4c8e-920c-0ba95c9b5ccc" containerName="revoke-keys" Nov 03 16:33:02.023398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.023384 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="00db8266-7b88-4c8e-920c-0ba95c9b5ccc" containerName="revoke-keys" Nov 03 16:33:02.027093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.027075 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.091289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.087918 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq"] Nov 03 16:33:02.124342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.124311 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.124602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.124575 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.124774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.124753 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdpq\" (UniqueName: \"kubernetes.io/projected/e44251e9-40b9-4e52-8ca7-14cbbb156036-kube-api-access-tvdpq\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.225582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.225547 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdpq\" (UniqueName: \"kubernetes.io/projected/e44251e9-40b9-4e52-8ca7-14cbbb156036-kube-api-access-tvdpq\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.225747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.225607 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.225747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.225728 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.226538 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.226435 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.228322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.228300 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.286868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.286841 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdpq\" (UniqueName: \"kubernetes.io/projected/e44251e9-40b9-4e52-8ca7-14cbbb156036-kube-api-access-tvdpq\") pod \"maas-api-revoke-sub-aab82594-ldbjq\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.340562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.340526 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:02.501061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.501028 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" Nov 03 16:33:02.501061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.501051 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm" event={"ID":"00db8266-7b88-4c8e-920c-0ba95c9b5ccc","Type":"ContainerDied","Data":"061f9c6526c6477da9b774ab4a961959a9d11bcfe90f6bb1157dad9df0e19727"} Nov 03 16:33:02.501553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.501131 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="061f9c6526c6477da9b774ab4a961959a9d11bcfe90f6bb1157dad9df0e19727" Nov 03 16:33:02.597385 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.597316 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:33:02.680767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.680742 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:33:02.736349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.736314 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq"] Nov 03 16:33:02.837132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.837097 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-openshift-ca\") pod \"409b02a4-2bb7-4514-b88f-6ae3356071b7\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " Nov 03 16:33:02.837475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.837445 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-xks-ca\") pod \"409b02a4-2bb7-4514-b88f-6ae3356071b7\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " Nov 03 16:33:02.837601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.837492 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409b02a4-2bb7-4514-b88f-6ae3356071b7-kube-api-access-m827j\" (UniqueName: \"kubernetes.io/projected/409b02a4-2bb7-4514-b88f-6ae3356071b7-kube-api-access-m827j\") pod \"409b02a4-2bb7-4514-b88f-6ae3356071b7\" (UID: \"409b02a4-2bb7-4514-b88f-6ae3356071b7\") " Nov 03 16:33:02.837601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.837571 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-openshift-ca" pod "409b02a4-2bb7-4514-b88f-6ae3356071b7" (UID: "409b02a4-2bb7-4514-b88f-6ae3356071b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:02.838031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.838007 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:02.914481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.914394 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-xks-ca" pod "409b02a4-2bb7-4514-b88f-6ae3356071b7" (UID: "409b02a4-2bb7-4514-b88f-6ae3356071b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:02.914481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.914440 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409b02a4-2bb7-4514-b88f-6ae3356071b7-kube-api-access-m827j" pod "409b02a4-2bb7-4514-b88f-6ae3356071b7" (UID: "409b02a4-2bb7-4514-b88f-6ae3356071b7"). InnerVolumeSpecName "kube-api-access-m827j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:02.939767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.939734 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409b02a4-2bb7-4514-b88f-6ae3356071b7-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:02.939767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:02.939767 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m827j\" (UniqueName: \"kubernetes.io/projected/409b02a4-2bb7-4514-b88f-6ae3356071b7-kube-api-access-m827j\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:03.531928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:03.531898 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" Nov 03 16:33:03.531928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:03.531912 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5" event={"ID":"409b02a4-2bb7-4514-b88f-6ae3356071b7","Type":"ContainerDied","Data":"e18684f467b25e3de8bc684e248fb7a3fe8b3ceebac9e23ba589491d8a9379d3"} Nov 03 16:33:03.532582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:03.531977 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e18684f467b25e3de8bc684e248fb7a3fe8b3ceebac9e23ba589491d8a9379d3" Nov 03 16:33:03.535184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:03.535158 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" event={"ID":"e44251e9-40b9-4e52-8ca7-14cbbb156036","Type":"ContainerStarted","Data":"889b630ee8731210359001627c1189f8725c523eef356c7b0271d692dbb9cfae"} Nov 03 16:33:03.535392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:03.535375 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" event={"ID":"e44251e9-40b9-4e52-8ca7-14cbbb156036","Type":"ContainerStarted","Data":"4c43860da7e6d500ed90782540007d4ab0648cc7d85ad35c6d7f63bc4b89b854"} Nov 03 16:33:03.670570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:03.662071 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" podStartSLOduration=2.6620523670000003 podStartE2EDuration="2.662052367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:03.655345054 +0000 UTC m=+1253.776397877" watchObservedRunningTime="2026-11-03 16:33:03.662052367 +0000 UTC m=+1253.783105192" Nov 03 16:33:04.596233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:04.595437 5946 generic.go:356] "Generic (PLEG): container finished" podID="e44251e9-40b9-4e52-8ca7-14cbbb156036" containerID="889b630ee8731210359001627c1189f8725c523eef356c7b0271d692dbb9cfae" exitCode=0 Nov 03 16:33:04.596233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:04.595654 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" event={"ID":"e44251e9-40b9-4e52-8ca7-14cbbb156036","Type":"ContainerDied","Data":"889b630ee8731210359001627c1189f8725c523eef356c7b0271d692dbb9cfae"} Nov 03 16:33:04.860582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:04.860483 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.036359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.036309 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.070355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.070301 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:33:05.326263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.319715 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.534234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.531349 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.637292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.636500 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.705277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.674298 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 16:33:05.705277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.674723 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.788258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.710729 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 16:33:05.788258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.733969 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:33:05.788258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.734020 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.788258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.783929 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.840255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.793075 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.861238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.860902 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:05.861439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:05.861249 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:33:06.092227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.087433 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 16:33:06.092227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.087379 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:06.138504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.138222 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:33:06.163587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.163561 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:33:06.329966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.325965 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:06.333435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.333375 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:33:06.594484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.535466 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:33:06.594484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.547597 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:06.643532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.643367 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 16:33:06.646894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.646871 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:33:06.675359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.675308 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:06.688062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.688031 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:33:06.736306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.736278 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:33:06.897450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.897335 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:06.898091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:06.898059 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 16:33:07.110026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.109993 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:33:07.179271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.179157 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:33:07.253304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.068200 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:33:07.253589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.253561 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:07.253909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.253888 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg"] Nov 03 16:33:07.259121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.259080 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" containerID="cri-o://3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b" gracePeriod=30 Nov 03 16:33:07.262057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.262033 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:33:07.406792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.406735 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:07.470361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.468648 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:33:07.612588 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.612542 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" event={"ID":"e44251e9-40b9-4e52-8ca7-14cbbb156036","Type":"ContainerDied","Data":"4c43860da7e6d500ed90782540007d4ab0648cc7d85ad35c6d7f63bc4b89b854"} Nov 03 16:33:07.612783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.612604 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c43860da7e6d500ed90782540007d4ab0648cc7d85ad35c6d7f63bc4b89b854" Nov 03 16:33:07.643229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.632398 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:07.685226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.685078 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:33:07.690961 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.690937 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:33:07.826231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.826171 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44251e9-40b9-4e52-8ca7-14cbbb156036-kube-api-access-tvdpq\" (UniqueName: \"kubernetes.io/projected/e44251e9-40b9-4e52-8ca7-14cbbb156036-kube-api-access-tvdpq\") pod \"e44251e9-40b9-4e52-8ca7-14cbbb156036\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " Nov 03 16:33:07.826742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.826704 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-openshift-ca\") pod \"e44251e9-40b9-4e52-8ca7-14cbbb156036\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " Nov 03 16:33:07.827057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.827036 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-xks-ca\") pod \"e44251e9-40b9-4e52-8ca7-14cbbb156036\" (UID: \"e44251e9-40b9-4e52-8ca7-14cbbb156036\") " Nov 03 16:33:07.828464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.828432 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-openshift-ca" pod "e44251e9-40b9-4e52-8ca7-14cbbb156036" (UID: "e44251e9-40b9-4e52-8ca7-14cbbb156036"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:07.890550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.890493 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44251e9-40b9-4e52-8ca7-14cbbb156036-kube-api-access-tvdpq" pod "e44251e9-40b9-4e52-8ca7-14cbbb156036" (UID: "e44251e9-40b9-4e52-8ca7-14cbbb156036"). InnerVolumeSpecName "kube-api-access-tvdpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:07.918325 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.918276 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-xks-ca" pod "e44251e9-40b9-4e52-8ca7-14cbbb156036" (UID: "e44251e9-40b9-4e52-8ca7-14cbbb156036"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:07.933995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.933906 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:07.933995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.933943 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44251e9-40b9-4e52-8ca7-14cbbb156036-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:07.933995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:07.933970 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvdpq\" (UniqueName: \"kubernetes.io/projected/e44251e9-40b9-4e52-8ca7-14cbbb156036-kube-api-access-tvdpq\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:08.624037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:08.624007 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq" Nov 03 16:33:10.245182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.245150 5946 scope.go:122] "RemoveContainer" containerID="720f869459e853ea13553a68ed62112337dd004d8d321ec2786ae7d89312b1cc" Nov 03 16:33:10.256116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.256091 5946 scope.go:122] "RemoveContainer" containerID="1e90c62fd222a9f9ff5a684d1f6674f9c37a5d1d504dc9131aadd01943280259" Nov 03 16:33:10.266543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.266515 5946 scope.go:122] "RemoveContainer" containerID="c72acec27ef36487e52c8f5aa915274341bd2d7d5cd8f99974b621de8ac15156" Nov 03 16:33:10.276282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.276262 5946 scope.go:122] "RemoveContainer" containerID="550bc001c6694ae44e1e4adc17ac4abd00bc935f8358ae121a6dc3510d592ec8" Nov 03 16:33:10.286443 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.286424 5946 scope.go:122] "RemoveContainer" containerID="aaf8c316ffcc392f53fc4f10d632ba8d27ae8419b23a8325e4550ebf9c257f47" Nov 03 16:33:10.295698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.295679 5946 scope.go:122] "RemoveContainer" containerID="9b888124d0d6d4bf661229281a1662cd2474938df2e2e0cce67e3981fd2b9951" Nov 03 16:33:10.311364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.311343 5946 scope.go:122] "RemoveContainer" containerID="34e1339d1c8bbf3f0c668f36facbe5e49e4c0c34589a095f471fd60346b9f8de" Nov 03 16:33:10.322043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.322016 5946 scope.go:122] "RemoveContainer" containerID="3c33f2e0760281055a5bfc2d71bf56c061be38f521d8f8989fa053016876866d" Nov 03 16:33:10.334066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.334039 5946 scope.go:122] "RemoveContainer" containerID="3c77dfbf27aeb5ee25f52d1fbb29697b959f5d16d76c6387a723cd71af83fa0e" Nov 03 16:33:10.724022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.723919 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc"] Nov 03 16:33:10.724820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.724795 5946 cpu_manager.go:414] "containerMap: removing container" podUID="409b02a4-2bb7-4514-b88f-6ae3356071b7" containerName="revoke-keys" Nov 03 16:33:10.724820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.724820 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409b02a4-2bb7-4514-b88f-6ae3356071b7" containerName="revoke-keys" Nov 03 16:33:10.725021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.724833 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e44251e9-40b9-4e52-8ca7-14cbbb156036" containerName="revoke-keys" Nov 03 16:33:10.725021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.724840 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44251e9-40b9-4e52-8ca7-14cbbb156036" containerName="revoke-keys" Nov 03 16:33:10.725021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.725002 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="409b02a4-2bb7-4514-b88f-6ae3356071b7" containerName="revoke-keys" Nov 03 16:33:10.725021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.725013 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44251e9-40b9-4e52-8ca7-14cbbb156036" containerName="revoke-keys" Nov 03 16:33:10.731366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.731342 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.773341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.773313 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.773507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.773369 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knm7p\" (UniqueName: \"kubernetes.io/projected/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-kube-api-access-knm7p\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.773507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.773479 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.787255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.787179 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc"] Nov 03 16:33:10.833329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.833264 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:33:10.833329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.833298 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:33:10.833641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.833340 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:33:10.836617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.836598 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:33:10.874616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.874587 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knm7p\" (UniqueName: \"kubernetes.io/projected/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-kube-api-access-knm7p\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.874776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.874641 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.874919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.874899 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.875529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.875508 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.877260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.877236 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:10.935774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:10.935741 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knm7p\" (UniqueName: \"kubernetes.io/projected/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-kube-api-access-knm7p\") pod \"maas-api-revoke-sub-bed516fd-gssfc\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:11.047481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:11.047443 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:11.436373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:11.436337 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc"] Nov 03 16:33:11.662266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:11.653638 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" event={"ID":"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3","Type":"ContainerStarted","Data":"31823f06657bdf04e59bb2e0374f064b813ae9df8091059bb576cf0bfbcea2fa"} Nov 03 16:33:11.763468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:11.763425 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:12.524882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.524853 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:33:12.595991 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:12.595952 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:33:12.599753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.599702 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-token\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.600195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.600130 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-podinfo\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.600437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.600376 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-credential-socket\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.600676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.600572 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-data\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.600676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.600647 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed1f329-b09b-4d11-b70f-79b1835336ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f329-b09b-4d11-b70f-79b1835336ed-istiod-ca-cert\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.600998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.600962 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-kube-api-access-z2dvj\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-kube-api-access-z2dvj\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.601229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.601157 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-envoy\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.601549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.601498 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-socket\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.601855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.601683 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-certs\") pod \"3ed1f329-b09b-4d11-b70f-79b1835336ed\" (UID: \"3ed1f329-b09b-4d11-b70f-79b1835336ed\") " Nov 03 16:33:12.602884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.602856 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-certs" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:12.603969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.603909 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-data" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:12.604182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.604150 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-socket" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:12.604272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.604233 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-credential-socket" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:12.604390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.604370 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed1f329-b09b-4d11-b70f-79b1835336ed-istiod-ca-cert" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:12.607170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.607112 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-envoy" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:12.607284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.607192 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-token" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:12.608845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.608809 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-podinfo" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:33:12.609082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.609058 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-kube-api-access-z2dvj" pod "3ed1f329-b09b-4d11-b70f-79b1835336ed" (UID: "3ed1f329-b09b-4d11-b70f-79b1835336ed"). InnerVolumeSpecName "kube-api-access-z2dvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:12.664070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.664038 5946 generic.go:356] "Generic (PLEG): container finished" podID="e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" containerID="8c2ac25ed462365671c0b02d4d2ff6391e297177f1e112df82473c3ac71dc358" exitCode=0 Nov 03 16:33:12.664271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.664116 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" event={"ID":"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3","Type":"ContainerDied","Data":"8c2ac25ed462365671c0b02d4d2ff6391e297177f1e112df82473c3ac71dc358"} Nov 03 16:33:12.665470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.665441 5946 generic.go:356] "Generic (PLEG): container finished" podID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerID="3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b" exitCode=0 Nov 03 16:33:12.665559 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.665483 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" Nov 03 16:33:12.665559 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.665484 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" event={"ID":"3ed1f329-b09b-4d11-b70f-79b1835336ed","Type":"ContainerDied","Data":"3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b"} Nov 03 16:33:12.665676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.665602 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg" event={"ID":"3ed1f329-b09b-4d11-b70f-79b1835336ed","Type":"ContainerDied","Data":"0cd7aa72621b604cfa2052ee8847b9aa15bfc0a1ee6e8704da3a938e5ac13869"} Nov 03 16:33:12.665676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.665636 5946 scope.go:122] "RemoveContainer" containerID="3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b" Nov 03 16:33:12.676782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.676765 5946 scope.go:122] "RemoveContainer" containerID="3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b" Nov 03 16:33:12.677030 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:12.677013 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b\": container with ID starting with 3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b not found: ID does not exist" containerID="3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b" Nov 03 16:33:12.677102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.677038 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b"} err="failed to get container status \"3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b\": rpc error: code = NotFound desc = could not find container \"3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b\": container with ID starting with 3ce21f075caeae8c9e1618f3b4fc1d474936591f8c500a2a05d7c35cb436a92b not found: ID does not exist" Nov 03 16:33:12.705170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705139 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705163 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705178 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705188 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705198 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed1f329-b09b-4d11-b70f-79b1835336ed-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705229 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dvj\" (UniqueName: \"kubernetes.io/projected/3ed1f329-b09b-4d11-b70f-79b1835336ed-kube-api-access-z2dvj\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705243 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705256 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.705508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.705268 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed1f329-b09b-4d11-b70f-79b1835336ed-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:12.806988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.806953 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp"] Nov 03 16:33:12.807570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.807552 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" Nov 03 16:33:12.807570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.807570 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" Nov 03 16:33:12.818035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.808224 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" containerName="istio-proxy" Nov 03 16:33:12.825482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.825444 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:12.871089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.871019 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp"] Nov 03 16:33:12.908549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.908509 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqpj\" (UniqueName: \"kubernetes.io/projected/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-kube-api-access-tdqpj\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:12.908731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.908706 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:12.908806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.908772 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:12.948319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.948290 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg"] Nov 03 16:33:12.954686 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:12.954659 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2vqg"] Nov 03 16:33:13.010429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.010391 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqpj\" (UniqueName: \"kubernetes.io/projected/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-kube-api-access-tdqpj\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:13.010623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.010528 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:13.010623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.010578 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:13.011313 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.011291 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:13.013117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.013099 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:13.073521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.073488 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqpj\" (UniqueName: \"kubernetes.io/projected/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-kube-api-access-tdqpj\") pod \"maas-api-revoke-sub-4a0c4eab-lg6fp\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:13.141569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.141491 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:13.525028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.524998 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp"] Nov 03 16:33:13.525406 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:13.525382 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f263ec_23f8_4cb9_a4f0_55969ca04b6b.slice/crio-b4936976e39edeff639ef578966269988809d3147494c33c5714dab5a3893177 WatchSource:0}: Error finding container b4936976e39edeff639ef578966269988809d3147494c33c5714dab5a3893177: Status 404 returned error can't find the container with id b4936976e39edeff639ef578966269988809d3147494c33c5714dab5a3893177 Nov 03 16:33:13.674399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.674361 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" event={"ID":"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b","Type":"ContainerStarted","Data":"b4936976e39edeff639ef578966269988809d3147494c33c5714dab5a3893177"} Nov 03 16:33:13.814074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.814042 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:13.923318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.923294 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-openshift-ca\") pod \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " Nov 03 16:33:13.923454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.923419 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-kube-api-access-knm7p\" (UniqueName: \"kubernetes.io/projected/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-kube-api-access-knm7p\") pod \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " Nov 03 16:33:13.923512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.923455 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-xks-ca\") pod \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\" (UID: \"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3\") " Nov 03 16:33:13.923727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.923699 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-openshift-ca" pod "e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" (UID: "e04904dd-d6d5-49b1-b2d5-39e41bc10fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:13.924165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.924098 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.925532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.925508 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-xks-ca" pod "e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" (UID: "e04904dd-d6d5-49b1-b2d5-39e41bc10fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:13.926108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:13.926085 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-kube-api-access-knm7p" pod "e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" (UID: "e04904dd-d6d5-49b1-b2d5-39e41bc10fc3"). InnerVolumeSpecName "kube-api-access-knm7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:14.006127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.006084 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed1f329-b09b-4d11-b70f-79b1835336ed" path="/var/lib/kubelet/pods/3ed1f329-b09b-4d11-b70f-79b1835336ed/volumes" Nov 03 16:33:14.025434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.025407 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knm7p\" (UniqueName: \"kubernetes.io/projected/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-kube-api-access-knm7p\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:14.025434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.025434 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:14.687108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.687064 5946 generic.go:356] "Generic (PLEG): container finished" podID="c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" containerID="8da43f51ed31dbc7c203cc6ad709231a6b1c7df9c544d4897762b67a791ee181" exitCode=0 Nov 03 16:33:14.687557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.687237 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" event={"ID":"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b","Type":"ContainerDied","Data":"8da43f51ed31dbc7c203cc6ad709231a6b1c7df9c544d4897762b67a791ee181"} Nov 03 16:33:14.689131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.689108 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" Nov 03 16:33:14.689131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.689119 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc" event={"ID":"e04904dd-d6d5-49b1-b2d5-39e41bc10fc3","Type":"ContainerDied","Data":"31823f06657bdf04e59bb2e0374f064b813ae9df8091059bb576cf0bfbcea2fa"} Nov 03 16:33:14.689395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:14.689172 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31823f06657bdf04e59bb2e0374f064b813ae9df8091059bb576cf0bfbcea2fa" Nov 03 16:33:15.102868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.102833 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7139-8cd802"] Nov 03 16:33:15.103476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.103459 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" containerName="revoke-keys" Nov 03 16:33:15.103538 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.103478 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" containerName="revoke-keys" Nov 03 16:33:15.103605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.103596 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" containerName="revoke-keys" Nov 03 16:33:15.107107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.107091 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:15.162621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.162597 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8nh5\"" Nov 03 16:33:15.164162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.164140 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7139-8cd802"] Nov 03 16:33:15.242488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.242455 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnb8\" (UniqueName: \"kubernetes.io/projected/b1229f05-6b05-4dda-b454-cc813615d60d-kube-api-access-ctnb8\") pod \"test-curl-7139-8cd802\" (UID: \"b1229f05-6b05-4dda-b454-cc813615d60d\") " pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:15.344200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.344165 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnb8\" (UniqueName: \"kubernetes.io/projected/b1229f05-6b05-4dda-b454-cc813615d60d-kube-api-access-ctnb8\") pod \"test-curl-7139-8cd802\" (UID: \"b1229f05-6b05-4dda-b454-cc813615d60d\") " pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:15.406653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.406576 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnb8\" (UniqueName: \"kubernetes.io/projected/b1229f05-6b05-4dda-b454-cc813615d60d-kube-api-access-ctnb8\") pod \"test-curl-7139-8cd802\" (UID: \"b1229f05-6b05-4dda-b454-cc813615d60d\") " pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:15.419227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.418912 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:15.813917 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:15.813886 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1229f05_6b05_4dda_b454_cc813615d60d.slice/crio-3bb59fa5e44789c8d49426196e5913e425b70a7cb5d4e517e39f18312e7777c4 WatchSource:0}: Error finding container 3bb59fa5e44789c8d49426196e5913e425b70a7cb5d4e517e39f18312e7777c4: Status 404 returned error can't find the container with id 3bb59fa5e44789c8d49426196e5913e425b70a7cb5d4e517e39f18312e7777c4 Nov 03 16:33:15.814705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.814680 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7139-8cd802"] Nov 03 16:33:15.816337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.816318 5946 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:33:15.856771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.856748 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:15.952671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.952573 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-kube-api-access-tdqpj\" (UniqueName: \"kubernetes.io/projected/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-kube-api-access-tdqpj\") pod \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " Nov 03 16:33:15.952859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.952689 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-xks-ca\") pod \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " Nov 03 16:33:15.952859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.952767 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-openshift-ca\") pod \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\" (UID: \"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b\") " Nov 03 16:33:15.953468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.953412 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-openshift-ca" pod "c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" (UID: "c6f263ec-23f8-4cb9-a4f0-55969ca04b6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:15.954975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.954949 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-kube-api-access-tdqpj" pod "c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" (UID: "c6f263ec-23f8-4cb9-a4f0-55969ca04b6b"). InnerVolumeSpecName "kube-api-access-tdqpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:15.955105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:15.955087 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-xks-ca" pod "c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" (UID: "c6f263ec-23f8-4cb9-a4f0-55969ca04b6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:16.055918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.055885 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqpj\" (UniqueName: \"kubernetes.io/projected/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-kube-api-access-tdqpj\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:16.055918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.055918 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:16.056180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.055936 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:16.559022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.558992 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:16.708617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.708507 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" Nov 03 16:33:16.708617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.708532 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp" event={"ID":"c6f263ec-23f8-4cb9-a4f0-55969ca04b6b","Type":"ContainerDied","Data":"b4936976e39edeff639ef578966269988809d3147494c33c5714dab5a3893177"} Nov 03 16:33:16.708617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.708586 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4936976e39edeff639ef578966269988809d3147494c33c5714dab5a3893177" Nov 03 16:33:16.710226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:16.710175 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7139-8cd802" event={"ID":"b1229f05-6b05-4dda-b454-cc813615d60d","Type":"ContainerStarted","Data":"3bb59fa5e44789c8d49426196e5913e425b70a7cb5d4e517e39f18312e7777c4"} Nov 03 16:33:18.003795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.003764 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:18.203143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.203113 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:18.227876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.227846 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r"] Nov 03 16:33:18.228549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.228532 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" containerName="revoke-keys" Nov 03 16:33:18.228605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.228551 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" containerName="revoke-keys" Nov 03 16:33:18.228662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.228653 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" containerName="revoke-keys" Nov 03 16:33:18.232303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.232287 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.292388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.292312 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r"] Nov 03 16:33:18.336733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.336699 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:33:18.336928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.336740 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:33:18.336928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.336844 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:33:18.337329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.337309 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:33:18.387928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.387890 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.388229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.388187 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlshs\" (UniqueName: \"kubernetes.io/projected/e62f356e-67cf-4a8c-b246-31996edeac03-kube-api-access-jlshs\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.388300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.388284 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.489415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.489382 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.489607 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.489528 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.489749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.489732 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlshs\" (UniqueName: \"kubernetes.io/projected/e62f356e-67cf-4a8c-b246-31996edeac03-kube-api-access-jlshs\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.490124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.490063 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.492246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.492220 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.550221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.550143 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlshs\" (UniqueName: \"kubernetes.io/projected/e62f356e-67cf-4a8c-b246-31996edeac03-kube-api-access-jlshs\") pod \"maas-api-revoke-sub-7834cd86-4bt8r\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:18.825061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.824984 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:18.848027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:18.847991 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:19.245494 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:19.245465 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62f356e_67cf_4a8c_b246_31996edeac03.slice/crio-9f7e64f65b1c1bc52e9596850b4491b081e86863394eaf74319af4f566d50909 WatchSource:0}: Error finding container 9f7e64f65b1c1bc52e9596850b4491b081e86863394eaf74319af4f566d50909: Status 404 returned error can't find the container with id 9f7e64f65b1c1bc52e9596850b4491b081e86863394eaf74319af4f566d50909 Nov 03 16:33:19.246708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:19.246685 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r"] Nov 03 16:33:19.748623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:19.748574 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" event={"ID":"e62f356e-67cf-4a8c-b246-31996edeac03","Type":"ContainerStarted","Data":"db6eac3d54eb12e277f65ae836f99dc7038d6363c2e8ec2f00d25bd75a271b66"} Nov 03 16:33:19.748825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:19.748641 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" event={"ID":"e62f356e-67cf-4a8c-b246-31996edeac03","Type":"ContainerStarted","Data":"9f7e64f65b1c1bc52e9596850b4491b081e86863394eaf74319af4f566d50909"} Nov 03 16:33:19.750280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:19.750257 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7139-8cd802" event={"ID":"b1229f05-6b05-4dda-b454-cc813615d60d","Type":"ContainerStarted","Data":"260b5c51aa1eb8ab61b33019c7d1ed8e23d36c94640627a7a63be667b9d8ce77"} Nov 03 16:33:19.874817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:19.874717 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" podStartSLOduration=1.8747030740000001 podStartE2EDuration="1.874703074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:19.87042823 +0000 UTC m=+1269.991481054" watchObservedRunningTime="2026-11-03 16:33:19.874703074 +0000 UTC m=+1269.995756033" Nov 03 16:33:19.989562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:19.989497 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7139-8cd802" podStartSLOduration=1.4959783309999999 podStartE2EDuration="4.989477722s" totalImagesPullingTime="3.493499391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:19.988367105 +0000 UTC m=+1270.109419927" watchObservedRunningTime="2026-11-03 16:33:19.989477722 +0000 UTC m=+1270.110530545" Nov 03 16:33:20.759438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:20.759402 5946 generic.go:356] "Generic (PLEG): container finished" podID="e62f356e-67cf-4a8c-b246-31996edeac03" containerID="db6eac3d54eb12e277f65ae836f99dc7038d6363c2e8ec2f00d25bd75a271b66" exitCode=0 Nov 03 16:33:20.759856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:20.759457 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" event={"ID":"e62f356e-67cf-4a8c-b246-31996edeac03","Type":"ContainerDied","Data":"db6eac3d54eb12e277f65ae836f99dc7038d6363c2e8ec2f00d25bd75a271b66"} Nov 03 16:33:21.365590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.365552 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:21.587439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.587405 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm"] Nov 03 16:33:21.591673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.591656 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.678818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.678739 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm"] Nov 03 16:33:21.732177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.732140 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.732355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.732191 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjwv\" (UniqueName: \"kubernetes.io/projected/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-kube-api-access-gcjwv\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.732355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.732318 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.834388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.834347 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.834813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.834438 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjwv\" (UniqueName: \"kubernetes.io/projected/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-kube-api-access-gcjwv\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.834813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.834503 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.835284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.835254 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.839947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.839921 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.896838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.896350 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjwv\" (UniqueName: \"kubernetes.io/projected/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-kube-api-access-gcjwv\") pod \"maas-api-revoke-sub-62592796-c5qnm\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.906066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.906036 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:21.952675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:21.952605 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:22.037512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.037478 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62f356e-67cf-4a8c-b246-31996edeac03-kube-api-access-jlshs\" (UniqueName: \"kubernetes.io/projected/e62f356e-67cf-4a8c-b246-31996edeac03-kube-api-access-jlshs\") pod \"e62f356e-67cf-4a8c-b246-31996edeac03\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " Nov 03 16:33:22.037690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.037628 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-openshift-ca\") pod \"e62f356e-67cf-4a8c-b246-31996edeac03\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " Nov 03 16:33:22.037769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.037750 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-xks-ca\") pod \"e62f356e-67cf-4a8c-b246-31996edeac03\" (UID: \"e62f356e-67cf-4a8c-b246-31996edeac03\") " Nov 03 16:33:22.038097 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.038064 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-openshift-ca" pod "e62f356e-67cf-4a8c-b246-31996edeac03" (UID: "e62f356e-67cf-4a8c-b246-31996edeac03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:22.038357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.038303 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:22.040121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.040096 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-xks-ca" pod "e62f356e-67cf-4a8c-b246-31996edeac03" (UID: "e62f356e-67cf-4a8c-b246-31996edeac03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:22.040121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.040097 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62f356e-67cf-4a8c-b246-31996edeac03-kube-api-access-jlshs" pod "e62f356e-67cf-4a8c-b246-31996edeac03" (UID: "e62f356e-67cf-4a8c-b246-31996edeac03"). InnerVolumeSpecName "kube-api-access-jlshs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:22.140727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.140698 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlshs\" (UniqueName: \"kubernetes.io/projected/e62f356e-67cf-4a8c-b246-31996edeac03-kube-api-access-jlshs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:22.140727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.140724 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62f356e-67cf-4a8c-b246-31996edeac03-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:22.294994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.294945 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm"] Nov 03 16:33:22.295119 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:22.295005 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa658d9_75a6_4e03_aa47_7d7266dd19eb.slice/crio-539d877ef6fe7402c402598768358929d66670d38503e0f58206f6f27b8c2ff6 WatchSource:0}: Error finding container 539d877ef6fe7402c402598768358929d66670d38503e0f58206f6f27b8c2ff6: Status 404 returned error can't find the container with id 539d877ef6fe7402c402598768358929d66670d38503e0f58206f6f27b8c2ff6 Nov 03 16:33:22.775573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.775539 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" Nov 03 16:33:22.775772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.775531 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r" event={"ID":"e62f356e-67cf-4a8c-b246-31996edeac03","Type":"ContainerDied","Data":"9f7e64f65b1c1bc52e9596850b4491b081e86863394eaf74319af4f566d50909"} Nov 03 16:33:22.775772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.775690 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7e64f65b1c1bc52e9596850b4491b081e86863394eaf74319af4f566d50909" Nov 03 16:33:22.777888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.777483 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" event={"ID":"0fa658d9-75a6-4e03-aa47-7d7266dd19eb","Type":"ContainerStarted","Data":"107d7a0052f2ea8be76ddada2b530ee8cd61277a89afb32855a0046b88290667"} Nov 03 16:33:22.777888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.777528 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" event={"ID":"0fa658d9-75a6-4e03-aa47-7d7266dd19eb","Type":"ContainerStarted","Data":"539d877ef6fe7402c402598768358929d66670d38503e0f58206f6f27b8c2ff6"} Nov 03 16:33:22.899094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:22.898995 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" podStartSLOduration=1.898981198 podStartE2EDuration="1.898981198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:22.897146566 +0000 UTC m=+1273.018199400" watchObservedRunningTime="2026-11-03 16:33:22.898981198 +0000 UTC m=+1273.020034053" Nov 03 16:33:23.065069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.065033 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb"] Nov 03 16:33:23.065694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.065678 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e62f356e-67cf-4a8c-b246-31996edeac03" containerName="revoke-keys" Nov 03 16:33:23.065759 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.065696 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62f356e-67cf-4a8c-b246-31996edeac03" containerName="revoke-keys" Nov 03 16:33:23.065832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.065823 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62f356e-67cf-4a8c-b246-31996edeac03" containerName="revoke-keys" Nov 03 16:33:23.069262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.069244 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.132090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.132057 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb"] Nov 03 16:33:23.152945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.152864 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.152945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.152902 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.153171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.153125 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwx54\" (UniqueName: \"kubernetes.io/projected/5dd837f4-986a-4b71-82f7-d58b77d78362-kube-api-access-gwx54\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.254775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.254741 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.254958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.254780 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.254958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.254881 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwx54\" (UniqueName: \"kubernetes.io/projected/5dd837f4-986a-4b71-82f7-d58b77d78362-kube-api-access-gwx54\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.255676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.255649 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.257361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.257343 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.315319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.315283 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwx54\" (UniqueName: \"kubernetes.io/projected/5dd837f4-986a-4b71-82f7-d58b77d78362-kube-api-access-gwx54\") pod \"maas-api-revoke-sub-99b4cecf-69bgb\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.383227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.383165 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:23.772025 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:23.771994 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd837f4_986a_4b71_82f7_d58b77d78362.slice/crio-d00367fa70681d5d4a2b696bef7bd968adb4f5de5f2785d8fc872f86f6806ced WatchSource:0}: Error finding container d00367fa70681d5d4a2b696bef7bd968adb4f5de5f2785d8fc872f86f6806ced: Status 404 returned error can't find the container with id d00367fa70681d5d4a2b696bef7bd968adb4f5de5f2785d8fc872f86f6806ced Nov 03 16:33:23.773053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.773030 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb"] Nov 03 16:33:23.794795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.794768 5946 generic.go:356] "Generic (PLEG): container finished" podID="0fa658d9-75a6-4e03-aa47-7d7266dd19eb" containerID="107d7a0052f2ea8be76ddada2b530ee8cd61277a89afb32855a0046b88290667" exitCode=0 Nov 03 16:33:23.794897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.794859 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" event={"ID":"0fa658d9-75a6-4e03-aa47-7d7266dd19eb","Type":"ContainerDied","Data":"107d7a0052f2ea8be76ddada2b530ee8cd61277a89afb32855a0046b88290667"} Nov 03 16:33:23.796424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:23.796399 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" event={"ID":"5dd837f4-986a-4b71-82f7-d58b77d78362","Type":"ContainerStarted","Data":"d00367fa70681d5d4a2b696bef7bd968adb4f5de5f2785d8fc872f86f6806ced"} Nov 03 16:33:24.805618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:24.805583 5946 generic.go:356] "Generic (PLEG): container finished" podID="5dd837f4-986a-4b71-82f7-d58b77d78362" containerID="aca505b3b99d3fd0e3c83d7c7dcfd48b001c8bda2dafa860ca048e801279dccf" exitCode=0 Nov 03 16:33:24.806068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:24.805659 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" event={"ID":"5dd837f4-986a-4b71-82f7-d58b77d78362","Type":"ContainerDied","Data":"aca505b3b99d3fd0e3c83d7c7dcfd48b001c8bda2dafa860ca048e801279dccf"} Nov 03 16:33:24.967848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:24.967826 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:25.016981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.016945 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59"] Nov 03 16:33:25.017524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.017510 5946 cpu_manager.go:414] "containerMap: removing container" podUID="0fa658d9-75a6-4e03-aa47-7d7266dd19eb" containerName="revoke-keys" Nov 03 16:33:25.017576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.017526 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa658d9-75a6-4e03-aa47-7d7266dd19eb" containerName="revoke-keys" Nov 03 16:33:25.017674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.017665 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa658d9-75a6-4e03-aa47-7d7266dd19eb" containerName="revoke-keys" Nov 03 16:33:25.021126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.021105 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.079706 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.079677 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-xks-ca\") pod \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " Nov 03 16:33:25.079866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.079725 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-kube-api-access-gcjwv\" (UniqueName: \"kubernetes.io/projected/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-kube-api-access-gcjwv\") pod \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " Nov 03 16:33:25.079866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.079767 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-openshift-ca\") pod \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\" (UID: \"0fa658d9-75a6-4e03-aa47-7d7266dd19eb\") " Nov 03 16:33:25.080185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.080153 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-openshift-ca" pod "0fa658d9-75a6-4e03-aa47-7d7266dd19eb" (UID: "0fa658d9-75a6-4e03-aa47-7d7266dd19eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:25.082039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.082009 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-xks-ca" pod "0fa658d9-75a6-4e03-aa47-7d7266dd19eb" (UID: "0fa658d9-75a6-4e03-aa47-7d7266dd19eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:25.082113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.082035 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-kube-api-access-gcjwv" pod "0fa658d9-75a6-4e03-aa47-7d7266dd19eb" (UID: "0fa658d9-75a6-4e03-aa47-7d7266dd19eb"). InnerVolumeSpecName "kube-api-access-gcjwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:25.117597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.117522 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59"] Nov 03 16:33:25.181142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181108 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181258 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181290 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181409 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkjb\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-kube-api-access-4zkjb\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181505 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181550 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181593 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181692 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181753 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b892a281-9708-45e3-b454-ce3d9abfd2cd-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.181944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181923 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:25.181982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181954 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcjwv\" (UniqueName: \"kubernetes.io/projected/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-kube-api-access-gcjwv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:25.181982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.181970 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa658d9-75a6-4e03-aa47-7d7266dd19eb-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:25.283116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283078 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283138 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkjb\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-kube-api-access-4zkjb\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283178 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283243 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283273 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283367 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283420 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b892a281-9708-45e3-b454-ce3d9abfd2cd-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283508 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283547 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283625 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283759 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.283894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283857 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.284006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.283976 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.284118 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.284104 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b892a281-9708-45e3-b454-ce3d9abfd2cd-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.285717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.285696 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.285982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.285964 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.343400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.343364 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.343529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.343436 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkjb\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-kube-api-access-4zkjb\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.636456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.636421 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:25.814642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.814615 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" Nov 03 16:33:25.815012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.814653 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm" event={"ID":"0fa658d9-75a6-4e03-aa47-7d7266dd19eb","Type":"ContainerDied","Data":"539d877ef6fe7402c402598768358929d66670d38503e0f58206f6f27b8c2ff6"} Nov 03 16:33:25.815012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.814703 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="539d877ef6fe7402c402598768358929d66670d38503e0f58206f6f27b8c2ff6" Nov 03 16:33:25.946724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:25.946690 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:26.078411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.078351 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:26.092738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.092703 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-xks-ca\") pod \"5dd837f4-986a-4b71-82f7-d58b77d78362\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " Nov 03 16:33:26.092917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.092785 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd837f4-986a-4b71-82f7-d58b77d78362-kube-api-access-gwx54\" (UniqueName: \"kubernetes.io/projected/5dd837f4-986a-4b71-82f7-d58b77d78362-kube-api-access-gwx54\") pod \"5dd837f4-986a-4b71-82f7-d58b77d78362\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " Nov 03 16:33:26.092994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.092923 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-openshift-ca\") pod \"5dd837f4-986a-4b71-82f7-d58b77d78362\" (UID: \"5dd837f4-986a-4b71-82f7-d58b77d78362\") " Nov 03 16:33:26.093609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.093577 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-openshift-ca" pod "5dd837f4-986a-4b71-82f7-d58b77d78362" (UID: "5dd837f4-986a-4b71-82f7-d58b77d78362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:26.095453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.095427 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-xks-ca" pod "5dd837f4-986a-4b71-82f7-d58b77d78362" (UID: "5dd837f4-986a-4b71-82f7-d58b77d78362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:26.095541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.095511 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd837f4-986a-4b71-82f7-d58b77d78362-kube-api-access-gwx54" pod "5dd837f4-986a-4b71-82f7-d58b77d78362" (UID: "5dd837f4-986a-4b71-82f7-d58b77d78362"). InnerVolumeSpecName "kube-api-access-gwx54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:26.129544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.127340 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59"] Nov 03 16:33:26.145782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.145681 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:26.146365 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.146330 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:26.146520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.146398 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:26.194666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.194638 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:26.194791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.194670 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwx54\" (UniqueName: \"kubernetes.io/projected/5dd837f4-986a-4b71-82f7-d58b77d78362-kube-api-access-gwx54\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:26.194791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.194683 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd837f4-986a-4b71-82f7-d58b77d78362-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:26.845757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.845729 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" Nov 03 16:33:26.846242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.845805 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb" event={"ID":"5dd837f4-986a-4b71-82f7-d58b77d78362","Type":"ContainerDied","Data":"d00367fa70681d5d4a2b696bef7bd968adb4f5de5f2785d8fc872f86f6806ced"} Nov 03 16:33:26.846242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.845899 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d00367fa70681d5d4a2b696bef7bd968adb4f5de5f2785d8fc872f86f6806ced" Nov 03 16:33:26.847734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.847712 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" event={"ID":"b892a281-9708-45e3-b454-ce3d9abfd2cd","Type":"ContainerStarted","Data":"bc2a74134eccf40d504b9105cc4d0ec36bd2d766aba42586bed953b0dd299b9e"} Nov 03 16:33:26.847835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:26.847753 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" event={"ID":"b892a281-9708-45e3-b454-ce3d9abfd2cd","Type":"ContainerStarted","Data":"523681b3b7754a577f324642be40a80c5390e25fdb98b1298acbc89039047b78"} Nov 03 16:33:27.024239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:27.024062 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:27.043332 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:27.043278 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podStartSLOduration=3.043259012 podStartE2EDuration="3.043259012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:27.022563737 +0000 UTC m=+1277.143616597" watchObservedRunningTime="2026-11-03 16:33:27.043259012 +0000 UTC m=+1277.164311835" Nov 03 16:33:27.240123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:27.240090 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:27.453803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:27.453766 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:27.639737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:27.639586 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:27.642563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:27.642183 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:33:27.642563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:27.642269 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 16:33:28.099842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.099808 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:28.546267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.546172 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:28.642266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.637557 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:33:28.642266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.637623 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 16:33:28.973703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.973614 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk"] Nov 03 16:33:28.974240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.974225 5946 cpu_manager.go:414] "containerMap: removing container" podUID="5dd837f4-986a-4b71-82f7-d58b77d78362" containerName="revoke-keys" Nov 03 16:33:28.974282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.974241 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd837f4-986a-4b71-82f7-d58b77d78362" containerName="revoke-keys" Nov 03 16:33:28.974386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.974377 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd837f4-986a-4b71-82f7-d58b77d78362" containerName="revoke-keys" Nov 03 16:33:28.978088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.978068 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:28.996369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.996340 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:28.996501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.996476 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:28.996551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:28.996524 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpq8r\" (UniqueName: \"kubernetes.io/projected/4589e023-68f5-417e-a830-a7324b300e7a-kube-api-access-rpq8r\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.048287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.048248 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk"] Nov 03 16:33:29.084671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.084635 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:33:29.084671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.084638 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:33:29.084671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.084649 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:33:29.086447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.086429 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:33:29.098389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.098350 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.098543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.098523 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.098631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.098589 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpq8r\" (UniqueName: \"kubernetes.io/projected/4589e023-68f5-417e-a830-a7324b300e7a-kube-api-access-rpq8r\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.099188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.098997 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.101445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.101422 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.160053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.160019 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpq8r\" (UniqueName: \"kubernetes.io/projected/4589e023-68f5-417e-a830-a7324b300e7a-kube-api-access-rpq8r\") pod \"maas-api-revoke-sub-167f28b4-92pnk\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.292962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.292919 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:29.640290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.637614 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:33:29.640290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.637679 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 16:33:29.740240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.723727 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk"] Nov 03 16:33:29.740240 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:29.726305 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4589e023_68f5_417e_a830_a7324b300e7a.slice/crio-8ae4bb49d05fd0b3e3331073c2d3ef1d4b92623aed411e362eb5dc3c97965451 WatchSource:0}: Error finding container 8ae4bb49d05fd0b3e3331073c2d3ef1d4b92623aed411e362eb5dc3c97965451: Status 404 returned error can't find the container with id 8ae4bb49d05fd0b3e3331073c2d3ef1d4b92623aed411e362eb5dc3c97965451 Nov 03 16:33:29.877551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:29.877514 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" event={"ID":"4589e023-68f5-417e-a830-a7324b300e7a","Type":"ContainerStarted","Data":"8ae4bb49d05fd0b3e3331073c2d3ef1d4b92623aed411e362eb5dc3c97965451"} Nov 03 16:33:30.642281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.638578 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:33:30.642281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.638654 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 16:33:30.866469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.866437 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8"] Nov 03 16:33:30.871084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.871065 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.888871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.888837 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" event={"ID":"4589e023-68f5-417e-a830-a7324b300e7a","Type":"ContainerStarted","Data":"6b055aeca602d55cdf0fc4b9776abdf720dec83a7cfd2887918ce45cb0c593c2"} Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.952636 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckpd\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-kube-api-access-wckpd\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.952717 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.952790 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.952819 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.952866 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.952928 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.953487 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.953544 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.954804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.953580 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:30.960037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:30.960011 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8"] Nov 03 16:33:31.055022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.054989 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055036 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055056 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055168 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckpd\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-kube-api-access-wckpd\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055199 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055256 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055273 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055298 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055338 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.055811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.055792 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.056464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.056433 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.056624 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.056601 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.056852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.056831 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.057060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.057025 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.058749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.058450 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.058749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.058529 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.139140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.139104 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.139502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.139404 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:31.141251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.141228 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckpd\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-kube-api-access-wckpd\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.158101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.158047 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" podStartSLOduration=3.158027512 podStartE2EDuration="3.158027512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:31.156649692 +0000 UTC m=+1281.277702527" watchObservedRunningTime="2026-11-03 16:33:31.158027512 +0000 UTC m=+1281.279080337" Nov 03 16:33:31.186944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.186865 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:31.544284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.544251 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7139-8cd802"] Nov 03 16:33:31.544473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.544442 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7139-8cd802" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" containerName="test-curl-7139-8cd802" containerID="cri-o://260b5c51aa1eb8ab61b33019c7d1ed8e23d36c94640627a7a63be667b9d8ce77" gracePeriod=2 Nov 03 16:33:31.551122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.551089 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7139-8cd802"] Nov 03 16:33:31.598056 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:31.598015 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7139-8cd802\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:31.638833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.638790 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:33:31.639085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.638851 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 16:33:31.639085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.638903 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:31.740429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.736444 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8"] Nov 03 16:33:31.745365 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.745334 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:31.745602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.745577 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:31.745737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.745716 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:31.750567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.750538 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7139-2d9001"] Nov 03 16:33:31.751461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.751438 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" containerName="test-curl-7139-8cd802" Nov 03 16:33:31.751548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.751465 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" containerName="test-curl-7139-8cd802" Nov 03 16:33:31.751663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.751651 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" containerName="test-curl-7139-8cd802" Nov 03 16:33:31.756841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.756825 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:31.811160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.810288 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7139-2d9001"] Nov 03 16:33:31.866467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.866433 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btx7\" (UniqueName: \"kubernetes.io/projected/6505df56-69a6-4103-ab68-60f2d405e12b-kube-api-access-7btx7\") pod \"test-curl-7139-2d9001\" (UID: \"6505df56-69a6-4103-ab68-60f2d405e12b\") " pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:31.884813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.880684 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9"] Nov 03 16:33:31.887542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.887515 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.932859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.913244 5946 generic.go:356] "Generic (PLEG): container finished" podID="4589e023-68f5-417e-a830-a7324b300e7a" containerID="6b055aeca602d55cdf0fc4b9776abdf720dec83a7cfd2887918ce45cb0c593c2" exitCode=0 Nov 03 16:33:31.933280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.932817 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" event={"ID":"4589e023-68f5-417e-a830-a7324b300e7a","Type":"ContainerDied","Data":"6b055aeca602d55cdf0fc4b9776abdf720dec83a7cfd2887918ce45cb0c593c2"} Nov 03 16:33:31.942792 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:31.942488 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7139-8cd802\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:31.958059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.958026 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:31.958997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.958968 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" event={"ID":"924e9362-8f6a-4bfa-b450-2427d1c22aa8","Type":"ContainerStarted","Data":"c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d"} Nov 03 16:33:31.959110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.959024 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" event={"ID":"924e9362-8f6a-4bfa-b450-2427d1c22aa8","Type":"ContainerStarted","Data":"7dd3dd030b92f407157590e51189db4d88d3fa9cf7db442c901ac0a4f597535a"} Nov 03 16:33:31.968174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.968097 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.968660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.968383 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btx7\" (UniqueName: \"kubernetes.io/projected/6505df56-69a6-4103-ab68-60f2d405e12b-kube-api-access-7btx7\") pod \"test-curl-7139-2d9001\" (UID: \"6505df56-69a6-4103-ab68-60f2d405e12b\") " pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:31.968660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.968455 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnslm\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-kube-api-access-wnslm\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.968660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.968490 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.968660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.968532 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.969827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.968917 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.969827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.969038 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d61d34-53f4-4839-8d58-e36c4506a6c2-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.969827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.969117 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.969827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.969237 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.969827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.969308 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:31.981841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:31.981814 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9"] Nov 03 16:33:31.994952 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:31.994916 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7139-8cd802\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:32.051285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.047556 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btx7\" (UniqueName: \"kubernetes.io/projected/6505df56-69a6-4103-ab68-60f2d405e12b-kube-api-access-7btx7\") pod \"test-curl-7139-2d9001\" (UID: \"6505df56-69a6-4103-ab68-60f2d405e12b\") " pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:32.070369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070248 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070586 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070618 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070683 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnslm\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-kube-api-access-wnslm\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070712 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070743 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070825 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070881 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d61d34-53f4-4839-8d58-e36c4506a6c2-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.070935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070927 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.071077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.070993 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.071284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.071264 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.071284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.071271 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.072090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.071691 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d61d34-53f4-4839-8d58-e36c4506a6c2-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.072090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.072006 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.074130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.074103 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:32.074261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.074189 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.074539 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.074525 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.150450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.150385 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnslm\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-kube-api-access-wnslm\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.154095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.153675 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.199337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.195741 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:32.199337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.195859 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:32.199337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.196029 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:32.199337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.196087 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:32.248096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.246520 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:32.392980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.391888 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:32.512267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.499251 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podStartSLOduration=2.499230186 podStartE2EDuration="2.499230186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:32.380445286 +0000 UTC m=+1282.501498111" watchObservedRunningTime="2026-11-03 16:33:32.499230186 +0000 UTC m=+1282.620283013" Nov 03 16:33:32.512267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.511990 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7139-2d9001"] Nov 03 16:33:32.580145 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:32.580059 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6505df56_69a6_4103_ab68_60f2d405e12b.slice/crio-6d84397d5b80d64f33e32ebe53e5d187b6fd07ffb3d0767eec145ec0bb1ac6c6 WatchSource:0}: Error finding container 6d84397d5b80d64f33e32ebe53e5d187b6fd07ffb3d0767eec145ec0bb1ac6c6: Status 404 returned error can't find the container with id 6d84397d5b80d64f33e32ebe53e5d187b6fd07ffb3d0767eec145ec0bb1ac6c6 Nov 03 16:33:32.642384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.641724 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:33:32.642384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.641801 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 16:33:32.747713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.747636 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9"] Nov 03 16:33:32.749799 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:32.748339 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d61d34_53f4_4839_8d58_e36c4506a6c2.slice/crio-71bcf38f6fe68c040d960ae0e07239e34b146b0a78502429d717980b88a67c0a WatchSource:0}: Error finding container 71bcf38f6fe68c040d960ae0e07239e34b146b0a78502429d717980b88a67c0a: Status 404 returned error can't find the container with id 71bcf38f6fe68c040d960ae0e07239e34b146b0a78502429d717980b88a67c0a Nov 03 16:33:32.762244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.760466 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:32.762244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.760672 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:32.762244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.760746 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:32.978955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.978907 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7139-2d9001" event={"ID":"6505df56-69a6-4103-ab68-60f2d405e12b","Type":"ContainerStarted","Data":"bb840b53df83f2f6ef58a269b1a8ae81d0e1a9955fd7163d2224c159a3eb8f01"} Nov 03 16:33:32.979129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.978965 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7139-2d9001" event={"ID":"6505df56-69a6-4103-ab68-60f2d405e12b","Type":"ContainerStarted","Data":"6d84397d5b80d64f33e32ebe53e5d187b6fd07ffb3d0767eec145ec0bb1ac6c6"} Nov 03 16:33:32.984448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.984411 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" event={"ID":"20d61d34-53f4-4839-8d58-e36c4506a6c2","Type":"ContainerStarted","Data":"d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53"} Nov 03 16:33:32.984615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:32.984470 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" event={"ID":"20d61d34-53f4-4839-8d58-e36c4506a6c2","Type":"ContainerStarted","Data":"71bcf38f6fe68c040d960ae0e07239e34b146b0a78502429d717980b88a67c0a"} Nov 03 16:33:33.102328 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.101601 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7139-2d9001" podStartSLOduration=2.101580256 podStartE2EDuration="2.101580256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:33.100477459 +0000 UTC m=+1283.221530283" watchObservedRunningTime="2026-11-03 16:33:33.101580256 +0000 UTC m=+1283.222633081" Nov 03 16:33:33.198252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.194096 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:33.198252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.194153 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:33.250536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.250428 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:33.261939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.261899 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:33.262306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.262185 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:33.276162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.276083 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podStartSLOduration=2.276059379 podStartE2EDuration="2.276059379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:33.240745769 +0000 UTC m=+1283.361798630" watchObservedRunningTime="2026-11-03 16:33:33.276059379 +0000 UTC m=+1283.397112204" Nov 03 16:33:33.338381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.334793 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:33.423796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.423761 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:33.510234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.505821 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-openshift-ca\") pod \"4589e023-68f5-417e-a830-a7324b300e7a\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " Nov 03 16:33:33.510234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.505939 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-xks-ca\") pod \"4589e023-68f5-417e-a830-a7324b300e7a\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " Nov 03 16:33:33.510234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.505989 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4589e023-68f5-417e-a830-a7324b300e7a-kube-api-access-rpq8r\" (UniqueName: \"kubernetes.io/projected/4589e023-68f5-417e-a830-a7324b300e7a-kube-api-access-rpq8r\") pod \"4589e023-68f5-417e-a830-a7324b300e7a\" (UID: \"4589e023-68f5-417e-a830-a7324b300e7a\") " Nov 03 16:33:33.510234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.508604 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-openshift-ca" pod "4589e023-68f5-417e-a830-a7324b300e7a" (UID: "4589e023-68f5-417e-a830-a7324b300e7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:33.551794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.551742 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-xks-ca" pod "4589e023-68f5-417e-a830-a7324b300e7a" (UID: "4589e023-68f5-417e-a830-a7324b300e7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:33.553193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.553119 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:33.561226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.560403 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4589e023-68f5-417e-a830-a7324b300e7a-kube-api-access-rpq8r" pod "4589e023-68f5-417e-a830-a7324b300e7a" (UID: "4589e023-68f5-417e-a830-a7324b300e7a"). InnerVolumeSpecName "kube-api-access-rpq8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:33.610532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.607902 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:33.610532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.607941 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4589e023-68f5-417e-a830-a7324b300e7a-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:33.610532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:33.607956 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpq8r\" (UniqueName: \"kubernetes.io/projected/4589e023-68f5-417e-a830-a7324b300e7a-kube-api-access-rpq8r\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:34.072412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.072377 5946 generic.go:356] "Generic (PLEG): container finished" podID="b1229f05-6b05-4dda-b454-cc813615d60d" containerID="260b5c51aa1eb8ab61b33019c7d1ed8e23d36c94640627a7a63be667b9d8ce77" exitCode=137 Nov 03 16:33:34.074477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.074447 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:34.085970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.085854 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" Nov 03 16:33:34.091087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.091038 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk" event={"ID":"4589e023-68f5-417e-a830-a7324b300e7a","Type":"ContainerDied","Data":"8ae4bb49d05fd0b3e3331073c2d3ef1d4b92623aed411e362eb5dc3c97965451"} Nov 03 16:33:34.091362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.091342 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ae4bb49d05fd0b3e3331073c2d3ef1d4b92623aed411e362eb5dc3c97965451" Nov 03 16:33:34.172257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.170735 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1229f05-6b05-4dda-b454-cc813615d60d-kube-api-access-ctnb8\" (UniqueName: \"kubernetes.io/projected/b1229f05-6b05-4dda-b454-cc813615d60d-kube-api-access-ctnb8\") pod \"b1229f05-6b05-4dda-b454-cc813615d60d\" (UID: \"b1229f05-6b05-4dda-b454-cc813615d60d\") " Nov 03 16:33:34.189618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.189578 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:34.189886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.189854 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:34.195452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.195420 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1229f05-6b05-4dda-b454-cc813615d60d-kube-api-access-ctnb8" pod "b1229f05-6b05-4dda-b454-cc813615d60d" (UID: "b1229f05-6b05-4dda-b454-cc813615d60d"). InnerVolumeSpecName "kube-api-access-ctnb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:34.249231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.248706 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:34.249231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.248784 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:34.273880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.273772 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctnb8\" (UniqueName: \"kubernetes.io/projected/b1229f05-6b05-4dda-b454-cc813615d60d-kube-api-access-ctnb8\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:34.305820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.305721 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:34.306162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.306101 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:34.308020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.307997 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:33:34.995446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:34.995417 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7139-2d9001"] Nov 03 16:33:35.002465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.002436 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7139-2d9001"] Nov 03 16:33:35.109043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.109008 5946 scope.go:122] "RemoveContainer" containerID="260b5c51aa1eb8ab61b33019c7d1ed8e23d36c94640627a7a63be667b9d8ce77" Nov 03 16:33:35.109736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.109049 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:35.110556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.110445 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7139-2d9001" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" containerName="test-curl-7139-2d9001" containerID="cri-o://bb840b53df83f2f6ef58a269b1a8ae81d0e1a9955fd7163d2224c159a3eb8f01" gracePeriod=2 Nov 03 16:33:35.166287 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:35.164500 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7139-2d9001\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:35.188563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.188526 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:35.188728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.188600 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:35.219736 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:35.219689 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7139-2d9001\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:35.251350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.251261 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:35.251681 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.251617 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:35.273221 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:35.272496 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7139-8cd802\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" pod="opendatahub/test-curl-7139-8cd802" Nov 03 16:33:35.330246 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:35.325506 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7139-2d9001\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:35.443032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.442840 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:35.666305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.666154 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:35.894498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:35.892737 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:36.108662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.108631 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:36.210234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.209732 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:36.210234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.209789 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:36.231000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.219570 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1229f05-6b05-4dda-b454-cc813615d60d" path="/var/lib/kubelet/pods/b1229f05-6b05-4dda-b454-cc813615d60d/volumes" Nov 03 16:33:36.252260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.250471 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:36.252260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.250544 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:36.324172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.324137 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:36.344746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.344709 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl"] Nov 03 16:33:36.358889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.345876 5946 cpu_manager.go:414] "containerMap: removing container" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" containerName="test-curl-7139-2d9001" Nov 03 16:33:36.359113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.359089 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" containerName="test-curl-7139-2d9001" Nov 03 16:33:36.359259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.359244 5946 cpu_manager.go:414] "containerMap: removing container" podUID="4589e023-68f5-417e-a830-a7324b300e7a" containerName="revoke-keys" Nov 03 16:33:36.359496 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.359480 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4589e023-68f5-417e-a830-a7324b300e7a" containerName="revoke-keys" Nov 03 16:33:36.359962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.359942 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" containerName="test-curl-7139-2d9001" Nov 03 16:33:36.360095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.360082 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="4589e023-68f5-417e-a830-a7324b300e7a" containerName="revoke-keys" Nov 03 16:33:36.367282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.366324 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:36.409853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.409810 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl"] Nov 03 16:33:36.466092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.464936 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e998bdb4-dockercfg-nbntz\"" Nov 03 16:33:36.466092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.465005 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e998bdb4\"" Nov 03 16:33:36.525153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.525112 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252s7\" (UniqueName: \"kubernetes.io/projected/1c69528a-235b-4f9b-a98b-7c8901709517-kube-api-access-252s7\") pod \"payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:36.525503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.525475 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69528a-235b-4f9b-a98b-7c8901709517-plugins-config-volume\") pod \"payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:36.634285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.628728 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252s7\" (UniqueName: \"kubernetes.io/projected/1c69528a-235b-4f9b-a98b-7c8901709517-kube-api-access-252s7\") pod \"payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:36.634285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.628815 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69528a-235b-4f9b-a98b-7c8901709517-plugins-config-volume\") pod \"payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:36.634285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.630093 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69528a-235b-4f9b-a98b-7c8901709517-plugins-config-volume\") pod \"payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:36.820227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.819970 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252s7\" (UniqueName: \"kubernetes.io/projected/1c69528a-235b-4f9b-a98b-7c8901709517-kube-api-access-252s7\") pod \"payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:36.908356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:36.902394 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:37.064865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.064832 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:37.210412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.203407 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:37.210412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.203488 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:37.282695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.275474 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:37.282695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.275540 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:37.289151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.288981 5946 generic.go:356] "Generic (PLEG): container finished" podID="6505df56-69a6-4103-ab68-60f2d405e12b" containerID="bb840b53df83f2f6ef58a269b1a8ae81d0e1a9955fd7163d2224c159a3eb8f01" exitCode=137 Nov 03 16:33:37.386874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.386834 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:37.593632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.592884 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl"] Nov 03 16:33:37.598480 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:33:37.598446 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c69528a_235b_4f9b_a98b_7c8901709517.slice/crio-a189b03f322eb1413bbaca679a098156da2e8993fe07720c4a1fc8db3a9d7f7f WatchSource:0}: Error finding container a189b03f322eb1413bbaca679a098156da2e8993fe07720c4a1fc8db3a9d7f7f: Status 404 returned error can't find the container with id a189b03f322eb1413bbaca679a098156da2e8993fe07720c4a1fc8db3a9d7f7f Nov 03 16:33:37.683311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.683285 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:37.797787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.797759 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6505df56-69a6-4103-ab68-60f2d405e12b-kube-api-access-7btx7\" (UniqueName: \"kubernetes.io/projected/6505df56-69a6-4103-ab68-60f2d405e12b-kube-api-access-7btx7\") pod \"6505df56-69a6-4103-ab68-60f2d405e12b\" (UID: \"6505df56-69a6-4103-ab68-60f2d405e12b\") " Nov 03 16:33:37.799903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.799878 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6505df56-69a6-4103-ab68-60f2d405e12b-kube-api-access-7btx7" pod "6505df56-69a6-4103-ab68-60f2d405e12b" (UID: "6505df56-69a6-4103-ab68-60f2d405e12b"). InnerVolumeSpecName "kube-api-access-7btx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:37.899456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:37.899411 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7btx7\" (UniqueName: \"kubernetes.io/projected/6505df56-69a6-4103-ab68-60f2d405e12b-kube-api-access-7btx7\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:38.007847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.007816 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6505df56-69a6-4103-ab68-60f2d405e12b" path="/var/lib/kubelet/pods/6505df56-69a6-4103-ab68-60f2d405e12b/volumes" Nov 03 16:33:38.068800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.068749 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:38.188012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.187966 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:38.188180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.188028 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:38.259233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.254902 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:38.259233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.254954 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:38.289568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.289536 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:38.298516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.298443 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" event={"ID":"1c69528a-235b-4f9b-a98b-7c8901709517","Type":"ContainerStarted","Data":"a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2"} Nov 03 16:33:38.298516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.298493 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" event={"ID":"1c69528a-235b-4f9b-a98b-7c8901709517","Type":"ContainerStarted","Data":"a189b03f322eb1413bbaca679a098156da2e8993fe07720c4a1fc8db3a9d7f7f"} Nov 03 16:33:38.299335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.299319 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:38.300475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.300454 5946 scope.go:122] "RemoveContainer" containerID="bb840b53df83f2f6ef58a269b1a8ae81d0e1a9955fd7163d2224c159a3eb8f01" Nov 03 16:33:38.300578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.300533 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7139-2d9001" Nov 03 16:33:38.446664 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.446591 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" podStartSLOduration=2.44657044 podStartE2EDuration="2.44657044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:38.420601506 +0000 UTC m=+1288.541654356" watchObservedRunningTime="2026-11-03 16:33:38.44657044 +0000 UTC m=+1288.567623264" Nov 03 16:33:38.489836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:38.489809 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:39.187829 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:39.187792 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:33:39.188011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:39.187856 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 16:33:39.247557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:39.247502 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:39.247752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:39.247561 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:40.193603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:40.193568 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:40.194130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:40.194094 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:40.194923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:40.194904 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:33:40.247522 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:40.247468 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:33:40.247713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:40.247523 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 16:33:41.262229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:41.257034 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:41.262229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:41.257773 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:41.262229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:41.259453 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:33:42.729237 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:33:42.719634 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:33:43.187232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.185740 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:43.187232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.185983 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 16:33:43.524235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.521757 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:43.524235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.522284 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 16:33:43.760234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.730694 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:43.760234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.755660 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:43.846150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.830571 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 16:33:43.901423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.847515 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 16:33:43.971232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.967503 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:43.996683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:43.996636 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.134076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.133972 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 16:33:44.192558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.192502 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.425282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.403404 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:33:44.543236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.533340 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.544418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.544381 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.569968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.569762 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.571304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.571281 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:33:44.616230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.604332 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.616230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.605109 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:33:44.827856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.827815 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 16:33:44.847167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.847111 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.852243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.849314 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:33:44.852598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.852578 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:33:44.866248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.828503 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.867505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.867457 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:44.874277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.870493 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:33:44.874277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.870575 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:33:44.925361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.925325 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:44.946558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.946515 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:33:44.946838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:44.946810 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:45.134565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.134470 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 16:33:45.135860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.135838 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:33:45.136924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.134809 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:45.223233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.197865 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:33:45.225057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.195304 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:45.304239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.258020 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:45.541235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.536395 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:45.541235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.536846 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:33:45.567387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.567342 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:33:45.583369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.583325 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 16:33:45.591803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.591771 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:33:45.620242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.609295 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:45.620242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.610401 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:45.620242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.611060 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:33:45.822289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.818669 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:45.837756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.837319 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 16:33:45.910340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.910297 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:45.917302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.916923 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 16:33:45.918010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:45.917707 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.138687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.138590 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.248245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.223765 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q"] Nov 03 16:33:46.288876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.288848 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:33:46.290262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.290243 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.307381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.307305 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.361131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.356848 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06f127ec-openshift-default-dockercfg-frnc8\"" Nov 03 16:33:46.361131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.360912 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q"] Nov 03 16:33:46.364315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.364193 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500458 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c959b-1822-45e4-a423-942e8af1fb26-istio-podinfo\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500523 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-envoy\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500664 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r87m2\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-kube-api-access-r87m2\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500694 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-socket\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500765 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-data\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500836 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-istio-token\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500858 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c959b-1822-45e4-a423-942e8af1fb26-istiod-ca-cert\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500913 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-credential-socket\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.500958 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-certs\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.652675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.634349 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 16:33:46.676253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.659329 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.718259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.700271 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-certs\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.961373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.718786 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:33:46.961373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.752748 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-certs\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.961373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.878490 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:46.961373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.880948 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 16:33:46.961373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.881978 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.961373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.911464 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.961373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.947403 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.962650 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.980540 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c959b-1822-45e4-a423-942e8af1fb26-istio-podinfo\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.980618 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-envoy\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.980797 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r87m2\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-kube-api-access-r87m2\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.980844 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-socket\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.980920 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-data\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.981008 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-istio-token\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.981038 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c959b-1822-45e4-a423-942e8af1fb26-istiod-ca-cert\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.981113 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-credential-socket\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:46.987234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:46.981673 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-credential-socket\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.016851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.016816 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-socket\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.067234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.032630 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-data\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.067234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.033424 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c959b-1822-45e4-a423-942e8af1fb26-istiod-ca-cert\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.139492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.139447 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:33:47.203956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.203925 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-envoy\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.299320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.295662 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 03 16:33:47.316229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.308499 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:47.365232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.362454 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 16:33:47.712327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.691197 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r87m2\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-kube-api-access-r87m2\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.728232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.725958 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-istio-token\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.728232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.726082 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c959b-1822-45e4-a423-942e8af1fb26-istio-podinfo\") pod \"e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.728232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.727330 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:47.733348 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.728825 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:47.901078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.900928 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:47.923500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.923348 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:47.955335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.955004 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 16:33:47.967733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.965333 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:47.967733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.966939 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:47.996247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.995832 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:48.004238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:47.996530 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:48.014932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.012314 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:33:48.147816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.109189 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:33:48.208409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.203466 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 16:33:48.208409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.203544 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 16:33:48.208409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.203826 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:33:48.208409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.205842 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:33:48.208409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.205892 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:33:48.264399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.263477 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:33:48.324240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.323622 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:48.508186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.503097 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:33:48.513404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.513359 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:48.514257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.513666 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:33:48.514257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.513676 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:48.514257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.513745 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:48.556460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.556427 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q"] Nov 03 16:33:48.714124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.714084 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" event={"ID":"f24c959b-1822-45e4-a423-942e8af1fb26","Type":"ContainerStarted","Data":"3de3cfeeb63776aa025476c4a03ad9f840b46d398312c3a9e588d543e46289fc"} Nov 03 16:33:48.749789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.749753 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft"] Nov 03 16:33:48.760336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.760311 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.827744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.826096 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e6def865-gw-openshift-default-dockercfg-n6vwh\"" Nov 03 16:33:48.867854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.847007 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-credential-socket\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.877856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.877821 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-token\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.878043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.872984 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:48.878147 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.878123 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-certs\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.878266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.878191 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-socket\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.878329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.878315 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vdf\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-kube-api-access-d2vdf\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.878394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.878363 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d12746a-0441-43a7-b3a0-18ce49c446ea-istiod-ca-cert\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.878533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.878508 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-data\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.878641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.871614 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft"] Nov 03 16:33:48.878745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.878726 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-podinfo\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.878835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.878819 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-envoy\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.990401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.990362 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-envoy\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.991088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.991055 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-credential-socket\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.991423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.991404 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-token\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.991638 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.991622 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-certs\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.991788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.991773 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-socket\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.991997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.991980 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vdf\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-kube-api-access-d2vdf\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.992645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.992510 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d12746a-0441-43a7-b3a0-18ce49c446ea-istiod-ca-cert\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.992987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.992966 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-socket\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.993120 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.992818 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-credential-socket\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.993241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.993035 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-data\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.993548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.993519 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-podinfo\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.993846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.992674 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-certs\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.994584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.994565 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-data\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:48.995105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:48.995084 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d12746a-0441-43a7-b3a0-18ce49c446ea-istiod-ca-cert\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:49.017333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.016022 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-envoy\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:49.017333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.016884 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-podinfo\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:49.055249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.055124 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-token\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:49.058366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.058292 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vdf\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-kube-api-access-d2vdf\") pod \"e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:49.201699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.191517 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:49.248362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.248036 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:49.648760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.647296 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft"] Nov 03 16:33:49.675977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.675938 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:49.676114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.676040 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:49.676114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.676088 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:33:49.729532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.729495 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" event={"ID":"f24c959b-1822-45e4-a423-942e8af1fb26","Type":"ContainerStarted","Data":"d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be"} Nov 03 16:33:49.731027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.730999 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" event={"ID":"3d12746a-0441-43a7-b3a0-18ce49c446ea","Type":"ContainerStarted","Data":"c57afe1c94249bdc0afc09a86610bc790c01fd73ec376d59db41947e942e8f8e"} Nov 03 16:33:49.733791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.733758 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:49.735102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.735075 5946 patch_prober.go:29] interesting pod/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:33:49.735187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.735131 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 16:33:49.865110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.865042 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podStartSLOduration=4.865022535 podStartE2EDuration="4.865022535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:49.858087997 +0000 UTC m=+1299.979140814" watchObservedRunningTime="2026-11-03 16:33:49.865022535 +0000 UTC m=+1299.986075359" Nov 03 16:33:49.896184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:49.896151 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:50.322395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:50.322370 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:33:50.739483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:50.739459 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:50.744137 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:50.744109 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" event={"ID":"3d12746a-0441-43a7-b3a0-18ce49c446ea","Type":"ContainerStarted","Data":"09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d"} Nov 03 16:33:50.744546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:50.744527 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:50.745433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:50.745413 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:33:51.123485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.123376 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podStartSLOduration=3.123360096 podStartE2EDuration="3.123360096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:51.118791725 +0000 UTC m=+1301.239844552" watchObservedRunningTime="2026-11-03 16:33:51.123360096 +0000 UTC m=+1301.244412916" Nov 03 16:33:51.198110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.198080 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:51.204004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.203978 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:51.452912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.452826 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8"] Nov 03 16:33:51.461189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.461165 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.512779 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.512742 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8"] Nov 03 16:33:51.519045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.518744 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:33:51.519045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.518878 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:33:51.519045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.519025 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:33:51.519324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.519088 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:33:51.572386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.572355 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.572573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.572500 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmq4q\" (UniqueName: \"kubernetes.io/projected/b5d9df82-06d5-4ef7-93d7-073786ca83aa-kube-api-access-xmq4q\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.572671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.572649 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.674056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.674006 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmq4q\" (UniqueName: \"kubernetes.io/projected/b5d9df82-06d5-4ef7-93d7-073786ca83aa-kube-api-access-xmq4q\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.674279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.674137 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.674342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.674332 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.674989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.674966 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.676742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.676716 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.735808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.735776 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmq4q\" (UniqueName: \"kubernetes.io/projected/b5d9df82-06d5-4ef7-93d7-073786ca83aa-kube-api-access-xmq4q\") pod \"maas-api-revoke-sub-2725e103-ssvl8\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:51.754055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.754021 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:51.755150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.755133 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:33:51.779258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:51.779226 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:52.169711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:52.169681 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8"] Nov 03 16:33:52.777872 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:52.777767 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" event={"ID":"b5d9df82-06d5-4ef7-93d7-073786ca83aa","Type":"ContainerStarted","Data":"4b7dc6b78152440cea89bb123cf93b0901740d624a10dd989a4237d48513767c"} Nov 03 16:33:52.778931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:52.777868 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" event={"ID":"b5d9df82-06d5-4ef7-93d7-073786ca83aa","Type":"ContainerStarted","Data":"0cb9a3b308da2835ed1cdbc2ca9558a212e039fcac652d5096e01396a080f5b0"} Nov 03 16:33:52.872969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:52.872940 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:52.908657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:52.908600 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" podStartSLOduration=1.908584575 podStartE2EDuration="1.908584575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:52.903545191 +0000 UTC m=+1303.024598033" watchObservedRunningTime="2026-11-03 16:33:52.908584575 +0000 UTC m=+1303.029637409" Nov 03 16:33:53.086493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:53.086412 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:53.768377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:53.768346 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:53.788842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:53.788798 5946 generic.go:356] "Generic (PLEG): container finished" podID="b5d9df82-06d5-4ef7-93d7-073786ca83aa" containerID="4b7dc6b78152440cea89bb123cf93b0901740d624a10dd989a4237d48513767c" exitCode=0 Nov 03 16:33:53.789362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:53.788900 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" event={"ID":"b5d9df82-06d5-4ef7-93d7-073786ca83aa","Type":"ContainerDied","Data":"4b7dc6b78152440cea89bb123cf93b0901740d624a10dd989a4237d48513767c"} Nov 03 16:33:53.986513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:53.986482 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:54.957317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:54.957286 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:55.030725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.030687 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-xks-ca\") pod \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " Nov 03 16:33:55.030917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.030871 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d9df82-06d5-4ef7-93d7-073786ca83aa-kube-api-access-xmq4q\" (UniqueName: \"kubernetes.io/projected/b5d9df82-06d5-4ef7-93d7-073786ca83aa-kube-api-access-xmq4q\") pod \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " Nov 03 16:33:55.030989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.030966 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-openshift-ca\") pod \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\" (UID: \"b5d9df82-06d5-4ef7-93d7-073786ca83aa\") " Nov 03 16:33:55.031311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.031285 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-openshift-ca" pod "b5d9df82-06d5-4ef7-93d7-073786ca83aa" (UID: "b5d9df82-06d5-4ef7-93d7-073786ca83aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:55.031668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.031640 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:55.032863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.032846 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-xks-ca" pod "b5d9df82-06d5-4ef7-93d7-073786ca83aa" (UID: "b5d9df82-06d5-4ef7-93d7-073786ca83aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:55.033110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.033090 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d9df82-06d5-4ef7-93d7-073786ca83aa-kube-api-access-xmq4q" pod "b5d9df82-06d5-4ef7-93d7-073786ca83aa" (UID: "b5d9df82-06d5-4ef7-93d7-073786ca83aa"). InnerVolumeSpecName "kube-api-access-xmq4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:55.133059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.132952 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmq4q\" (UniqueName: \"kubernetes.io/projected/b5d9df82-06d5-4ef7-93d7-073786ca83aa-kube-api-access-xmq4q\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:55.133059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.132985 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9df82-06d5-4ef7-93d7-073786ca83aa-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:55.164194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.164163 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:55.807269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.807238 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" Nov 03 16:33:55.807269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.807259 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8" event={"ID":"b5d9df82-06d5-4ef7-93d7-073786ca83aa","Type":"ContainerDied","Data":"0cb9a3b308da2835ed1cdbc2ca9558a212e039fcac652d5096e01396a080f5b0"} Nov 03 16:33:55.807519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:55.807310 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb9a3b308da2835ed1cdbc2ca9558a212e039fcac652d5096e01396a080f5b0" Nov 03 16:33:56.369909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:56.369876 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:56.560094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:33:56.560062 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:00.165684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:00.162632 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:01.515400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:01.515369 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:01.758761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:01.758192 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:02.213800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.213772 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26"] Nov 03 16:34:02.214637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.214611 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b5d9df82-06d5-4ef7-93d7-073786ca83aa" containerName="revoke-keys" Nov 03 16:34:02.214637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.214636 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d9df82-06d5-4ef7-93d7-073786ca83aa" containerName="revoke-keys" Nov 03 16:34:02.214845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.214830 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d9df82-06d5-4ef7-93d7-073786ca83aa" containerName="revoke-keys" Nov 03 16:34:02.223230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.220807 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.285276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.285243 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26"] Nov 03 16:34:02.300111 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.300074 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.300366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.300302 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkdb\" (UniqueName: \"kubernetes.io/projected/183e7f2b-d882-498a-a85a-2a2a0a88025f-kube-api-access-8lkdb\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.300450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.300372 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.301943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.301897 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:34:02.302350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.302332 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:34:02.302748 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.302631 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:34:02.328377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.328348 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:34:02.401965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.401929 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.402138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.402002 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkdb\" (UniqueName: \"kubernetes.io/projected/183e7f2b-d882-498a-a85a-2a2a0a88025f-kube-api-access-8lkdb\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.402138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.402038 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.402596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.402574 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.404530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.404507 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.462613 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.462578 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkdb\" (UniqueName: \"kubernetes.io/projected/183e7f2b-d882-498a-a85a-2a2a0a88025f-kube-api-access-8lkdb\") pod \"maas-api-revoke-sub-83f1fd9e-64h26\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.579453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.579413 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:02.986455 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:34:02.986423 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183e7f2b_d882_498a_a85a_2a2a0a88025f.slice/crio-148da6864dfbae9794e9f6ec839cd071bbae4dca960db99760329c284acb82e3 WatchSource:0}: Error finding container 148da6864dfbae9794e9f6ec839cd071bbae4dca960db99760329c284acb82e3: Status 404 returned error can't find the container with id 148da6864dfbae9794e9f6ec839cd071bbae4dca960db99760329c284acb82e3 Nov 03 16:34:02.987727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:02.987703 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26"] Nov 03 16:34:03.525768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:03.525733 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:03.886612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:03.886569 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" event={"ID":"183e7f2b-d882-498a-a85a-2a2a0a88025f","Type":"ContainerStarted","Data":"c05fd76850c91d087a393d60f543bb9c889c9a9ef9466352e308d2a214e9348d"} Nov 03 16:34:03.887070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:03.886629 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" event={"ID":"183e7f2b-d882-498a-a85a-2a2a0a88025f","Type":"ContainerStarted","Data":"148da6864dfbae9794e9f6ec839cd071bbae4dca960db99760329c284acb82e3"} Nov 03 16:34:04.007899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:04.007849 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" podStartSLOduration=2.007835389 podStartE2EDuration="2.007835389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:04.006289619 +0000 UTC m=+1314.127342443" watchObservedRunningTime="2026-11-03 16:34:04.007835389 +0000 UTC m=+1314.128888211" Nov 03 16:34:04.896697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:04.896666 5946 generic.go:356] "Generic (PLEG): container finished" podID="183e7f2b-d882-498a-a85a-2a2a0a88025f" containerID="c05fd76850c91d087a393d60f543bb9c889c9a9ef9466352e308d2a214e9348d" exitCode=0 Nov 03 16:34:04.897103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:04.896760 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" event={"ID":"183e7f2b-d882-498a-a85a-2a2a0a88025f","Type":"ContainerDied","Data":"c05fd76850c91d087a393d60f543bb9c889c9a9ef9466352e308d2a214e9348d"} Nov 03 16:34:06.064378 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.064350 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:06.164882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.164801 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-xks-ca\") pod \"183e7f2b-d882-498a-a85a-2a2a0a88025f\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " Nov 03 16:34:06.164882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.164879 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-openshift-ca\") pod \"183e7f2b-d882-498a-a85a-2a2a0a88025f\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " Nov 03 16:34:06.165234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.164906 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183e7f2b-d882-498a-a85a-2a2a0a88025f-kube-api-access-8lkdb\" (UniqueName: \"kubernetes.io/projected/183e7f2b-d882-498a-a85a-2a2a0a88025f-kube-api-access-8lkdb\") pod \"183e7f2b-d882-498a-a85a-2a2a0a88025f\" (UID: \"183e7f2b-d882-498a-a85a-2a2a0a88025f\") " Nov 03 16:34:06.165314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.165260 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-openshift-ca" pod "183e7f2b-d882-498a-a85a-2a2a0a88025f" (UID: "183e7f2b-d882-498a-a85a-2a2a0a88025f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:06.165719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.165695 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:06.167141 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.167117 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-xks-ca" pod "183e7f2b-d882-498a-a85a-2a2a0a88025f" (UID: "183e7f2b-d882-498a-a85a-2a2a0a88025f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:06.167318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.167298 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183e7f2b-d882-498a-a85a-2a2a0a88025f-kube-api-access-8lkdb" pod "183e7f2b-d882-498a-a85a-2a2a0a88025f" (UID: "183e7f2b-d882-498a-a85a-2a2a0a88025f"). InnerVolumeSpecName "kube-api-access-8lkdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:06.267014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.266984 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183e7f2b-d882-498a-a85a-2a2a0a88025f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:06.267014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.267013 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkdb\" (UniqueName: \"kubernetes.io/projected/183e7f2b-d882-498a-a85a-2a2a0a88025f-kube-api-access-8lkdb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:06.915307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.915280 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" Nov 03 16:34:06.915491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.915281 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26" event={"ID":"183e7f2b-d882-498a-a85a-2a2a0a88025f","Type":"ContainerDied","Data":"148da6864dfbae9794e9f6ec839cd071bbae4dca960db99760329c284acb82e3"} Nov 03 16:34:06.915491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:06.915400 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="148da6864dfbae9794e9f6ec839cd071bbae4dca960db99760329c284acb82e3" Nov 03 16:34:07.247716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:07.247685 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:09.882812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:09.882778 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:10.424021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.423988 5946 scope.go:122] "RemoveContainer" containerID="17df887a069e6ea600b8c05a6d90ac38487db509c3b2b4064cc77d397bef3879" Nov 03 16:34:10.435794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.435773 5946 scope.go:122] "RemoveContainer" containerID="a660a2ee82413319f99d405214a93bde361e819196b3e433eae836dc263f7a00" Nov 03 16:34:10.445744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.445728 5946 scope.go:122] "RemoveContainer" containerID="020b7fe15421a2694abe3c6c9ea72ea2a0274e8f587cf2db1c6dd3a9069d9adf" Nov 03 16:34:10.455687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.455665 5946 scope.go:122] "RemoveContainer" containerID="b1d54c8692b07ce93b9b669345b8a8b36906e7b41ad644d073188409e23c02a1" Nov 03 16:34:10.468737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.468709 5946 scope.go:122] "RemoveContainer" containerID="898c93f24b29721e2feb1c7fe452d3881317cad690921dcbc12f2087f3c1ba1c" Nov 03 16:34:10.478983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.478961 5946 scope.go:122] "RemoveContainer" containerID="0977a68122c8f4f4baa4a1296506f91428f0eef865e82ccfa63b851624d2e9c4" Nov 03 16:34:10.489068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.489048 5946 scope.go:122] "RemoveContainer" containerID="8bd0561007a3197340cbff60e61f06bb49a97dac9b70120f1b1aaa583a67512d" Nov 03 16:34:10.498734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.498715 5946 scope.go:122] "RemoveContainer" containerID="2ea4a09d1501f82682587604f2d60b506f66533fd8c14d05b10343e9d30f112d" Nov 03 16:34:10.508557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.508537 5946 scope.go:122] "RemoveContainer" containerID="4ebc8d9f72c1f44ca6d1fcee82ff51f83733dd2c00f279a6a6a87dd64b58f9f3" Nov 03 16:34:10.518400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:10.518383 5946 scope.go:122] "RemoveContainer" containerID="6e46869a9baa6c8609be3c1ee341a8b1d03205898068cfde267389a89d8b4048" Nov 03 16:34:11.068670 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:11.068640 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:12.153965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:12.153932 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:12.586546 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:34:12.586509 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:34:13.893779 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:13.893723 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:34:14.569232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:14.568287 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:14.803227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:14.803166 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:15.480231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.472984 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:15.768301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.767273 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr"] Nov 03 16:34:15.768473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.768374 5946 cpu_manager.go:414] "containerMap: removing container" podUID="183e7f2b-d882-498a-a85a-2a2a0a88025f" containerName="revoke-keys" Nov 03 16:34:15.768473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.768397 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183e7f2b-d882-498a-a85a-2a2a0a88025f" containerName="revoke-keys" Nov 03 16:34:15.768604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.768591 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="183e7f2b-d882-498a-a85a-2a2a0a88025f" containerName="revoke-keys" Nov 03 16:34:15.793233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.790381 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.821590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.817649 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c"] Nov 03 16:34:15.847235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.845749 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:15.847235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.847164 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr"] Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884381 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884447 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884499 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884557 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ccd\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-kube-api-access-s7ccd\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884698 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884731 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884765 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884846 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.886714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.884883 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:15.920616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.920564 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:15.927254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.924583 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:15.979266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:15.968297 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c"] Nov 03 16:34:16.347235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.342080 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:16.347235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.342948 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 16:34:16.347235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.343392 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 16:34:16.608268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.535366 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:16.652359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.638431 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:34:16.669424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.669370 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:16.673667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.673624 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715443 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715516 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715570 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715661 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715719 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715763 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715805 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715868 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715915 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.715993 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.716074 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.716123 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.730595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.716191 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.758363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.745741 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.769682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.769642 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:16.960293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.958339 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 16:34:16.987184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.986529 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:34:16.987184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.986657 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:16.988026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:16.988000 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:34:17.007697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.005993 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.031676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.008113 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb"] Nov 03 16:34:17.032003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.020640 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.032793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.030360 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:17.033052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.030522 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.033266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.031220 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.061137 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.045707 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.064146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.062566 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.064146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.062665 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ccd\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-kube-api-access-s7ccd\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.064146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.062715 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lff\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-kube-api-access-c6lff\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.064146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.062772 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.100745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.100681 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.108629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.108598 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172460 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172535 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172590 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172628 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172702 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172763 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172863 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172932 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lff\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-kube-api-access-c6lff\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.172979 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.174115 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.177928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.176641 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.185947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.185916 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.186363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.186343 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.195286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.193944 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9"] Nov 03 16:34:17.199086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.198426 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.201575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.201560 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:34:17.201736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.201723 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5"] Nov 03 16:34:17.205784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.205766 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.218706 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.218641 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb"] Nov 03 16:34:17.219033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.219013 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9"] Nov 03 16:34:17.219572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.219545 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5"] Nov 03 16:34:17.219736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.218947 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.246455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.246388 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.247028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.247003 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.247907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.247882 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ccd\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-kube-api-access-s7ccd\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.257533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.257509 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.266337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.265756 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-161f6056-openshift-default-dockercfg-t2hfs\"" Nov 03 16:34:17.267629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.267561 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lff\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-kube-api-access-c6lff\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.268314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.268180 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279241 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-podinfo\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279297 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e5925d-2e74-4a38-a86b-5bbb45424268-istiod-ca-cert\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279326 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e16998-b28e-4e06-b395-74853a18ce79-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279365 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279391 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279423 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279454 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e16998-b28e-4e06-b395-74853a18ce79-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279495 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279577 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-certs\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279615 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcfj\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-kube-api-access-8hcfj\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279643 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279673 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279708 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279733 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279762 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279848 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-data\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279874 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279899 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dzb\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-kube-api-access-g5dzb\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.279949 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-credential-socket\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280019 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-envoy\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280124 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-token\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280154 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8vp\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-kube-api-access-6z8vp\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280196 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280250 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280284 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280374 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-socket\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.285269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.280410 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.323254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.323201 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.373232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.366465 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:17.373232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.367286 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:34:17.373232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.369832 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:17.373232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.370566 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.384736 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-socket\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.384788 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.384842 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-podinfo\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.384872 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e5925d-2e74-4a38-a86b-5bbb45424268-istiod-ca-cert\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.384901 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e16998-b28e-4e06-b395-74853a18ce79-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.384940 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.384965 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385000 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385034 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e16998-b28e-4e06-b395-74853a18ce79-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385076 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385161 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-certs\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385200 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcfj\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-kube-api-access-8hcfj\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385248 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385276 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385309 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385333 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385366 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385458 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-data\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385501 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385526 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dzb\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-kube-api-access-g5dzb\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385579 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-credential-socket\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385646 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-envoy\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385740 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-token\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385766 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8vp\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-kube-api-access-6z8vp\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385805 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385837 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.385870 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.386465 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.386706 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.388324 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-socket\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.388687 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.389114 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.390660 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.391005 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-certs\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.391897 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e5925d-2e74-4a38-a86b-5bbb45424268-istiod-ca-cert\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.392410 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.393100 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e16998-b28e-4e06-b395-74853a18ce79-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.394293 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-data\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.394531 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.395009 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-credential-socket\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.395639 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e16998-b28e-4e06-b395-74853a18ce79-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.396017 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.396560 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.397627 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.397658 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.397789 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-envoy\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.399232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.398760 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.409673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.409600 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.411760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.411735 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-podinfo\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.436931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.436663 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:17.545031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.544999 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8vp\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-kube-api-access-6z8vp\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.554241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.552749 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:17.554894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.554867 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:17.578895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.578867 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:34:17.704234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.700130 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dzb\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-kube-api-access-g5dzb\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.711287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.711252 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcfj\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-kube-api-access-8hcfj\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.737447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.737381 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-token\") pod \"e2e-derive-161f6056-openshift-default-c456fd685-trmm9\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.777245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.765826 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:17.811294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.796515 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:17.831113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:17.812101 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:18.255238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.255128 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:18.548319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.548284 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c"] Nov 03 16:34:18.559810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.559770 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:18.560169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.560130 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:18.560439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.560412 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:18.562744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.562716 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:18.617658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.581256 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr"] Nov 03 16:34:18.617658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.615935 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:18.617658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:18.616034 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.005224 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:34:19.005164 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c429cbc_c372_4d6e_b5f3_69e8374d8292.slice/crio-01b2f77bceae326520b6a17340c98df44d1e66966081b57761914f792d19b2d1 WatchSource:0}: Error finding container 01b2f77bceae326520b6a17340c98df44d1e66966081b57761914f792d19b2d1: Status 404 returned error can't find the container with id 01b2f77bceae326520b6a17340c98df44d1e66966081b57761914f792d19b2d1 Nov 03 16:34:19.007373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.007342 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5"] Nov 03 16:34:19.009005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.008963 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.009114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.009054 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.009114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.009094 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.041432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.041380 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9"] Nov 03 16:34:19.047403 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:34:19.047350 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e16998_b28e_4e06_b395_74853a18ce79.slice/crio-e9dc2f3a1c7925b24051259258abed6ac257a72fb03b60434a56b574bdba1a44 WatchSource:0}: Error finding container e9dc2f3a1c7925b24051259258abed6ac257a72fb03b60434a56b574bdba1a44: Status 404 returned error can't find the container with id e9dc2f3a1c7925b24051259258abed6ac257a72fb03b60434a56b574bdba1a44 Nov 03 16:34:19.051472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.050371 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.051472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.050475 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.051472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.050521 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.062287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.057309 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb"] Nov 03 16:34:19.063971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.063922 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.064086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.064034 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.064086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.064078 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:19.636804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.636768 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" event={"ID":"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430","Type":"ContainerStarted","Data":"2b0cfcf3ab7da0b1497b29efd71ef4fc56bdb88610171944d8826346491e15bb"} Nov 03 16:34:19.637818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.637779 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" event={"ID":"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430","Type":"ContainerStarted","Data":"2bda2cb39b70f8e9fdb79256686bbf85cdf36af210661676ef5e29897751ac51"} Nov 03 16:34:19.643615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.639188 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" event={"ID":"d6e16998-b28e-4e06-b395-74853a18ce79","Type":"ContainerStarted","Data":"d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588"} Nov 03 16:34:19.643615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.639285 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" event={"ID":"d6e16998-b28e-4e06-b395-74853a18ce79","Type":"ContainerStarted","Data":"e9dc2f3a1c7925b24051259258abed6ac257a72fb03b60434a56b574bdba1a44"} Nov 03 16:34:19.643615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.642128 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" event={"ID":"2c429cbc-c372-4d6e-b5f3-69e8374d8292","Type":"ContainerStarted","Data":"58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e"} Nov 03 16:34:19.643615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.642160 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" event={"ID":"2c429cbc-c372-4d6e-b5f3-69e8374d8292","Type":"ContainerStarted","Data":"01b2f77bceae326520b6a17340c98df44d1e66966081b57761914f792d19b2d1"} Nov 03 16:34:19.644492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.644417 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" event={"ID":"f8e5925d-2e74-4a38-a86b-5bbb45424268","Type":"ContainerStarted","Data":"6a019ba89d568f8299b1030aa2adc8349f53f36b4f8f0b87b8d5c69b677e2c98"} Nov 03 16:34:19.644492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.644458 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" event={"ID":"f8e5925d-2e74-4a38-a86b-5bbb45424268","Type":"ContainerStarted","Data":"192dc804bcf6f6be3226c2733c4eb04c31a32aee20019b9db63cb0b5d9c6daa4"} Nov 03 16:34:19.646967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.646521 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" event={"ID":"e589bc27-2ffb-4627-8dc4-2ed8a606e716","Type":"ContainerStarted","Data":"ec2d4d85b8cf86717213016776b05f9f65c1c63225e2cbdf4bac0d60d6151b3e"} Nov 03 16:34:19.646967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.646554 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" event={"ID":"e589bc27-2ffb-4627-8dc4-2ed8a606e716","Type":"ContainerStarted","Data":"a50331422ed83e75f872948a2bb66a6b4ec903e31457561aaed975d11439a892"} Nov 03 16:34:19.790227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.790098 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podStartSLOduration=4.790077752 podStartE2EDuration="4.790077752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:19.77729681 +0000 UTC m=+1329.898349644" watchObservedRunningTime="2026-11-03 16:34:19.790077752 +0000 UTC m=+1329.911130574" Nov 03 16:34:19.815546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.815511 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:19.815830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:19.815812 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:20.017233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.013969 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podStartSLOduration=5.013944493 podStartE2EDuration="5.013944493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:19.951515091 +0000 UTC m=+1330.072567918" watchObservedRunningTime="2026-11-03 16:34:20.013944493 +0000 UTC m=+1330.134997316" Nov 03 16:34:20.308293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.299327 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:20.308293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.301357 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:20.308293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.301402 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:20.464873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.464840 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:20.492383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.492342 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:20.576465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.515701 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podStartSLOduration=4.5156780439999995 podStartE2EDuration="4.515678044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:20.49309069 +0000 UTC m=+1330.614143515" watchObservedRunningTime="2026-11-03 16:34:20.515678044 +0000 UTC m=+1330.636730867" Nov 03 16:34:20.576986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.576915 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podStartSLOduration=4.57689193 podStartE2EDuration="4.57689193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:20.205458576 +0000 UTC m=+1330.326511403" watchObservedRunningTime="2026-11-03 16:34:20.57689193 +0000 UTC m=+1330.697944755" Nov 03 16:34:20.603258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.562640 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:20.603258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.595407 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:20.605054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.604824 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:20.605054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.604884 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:20.912287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.868000 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podStartSLOduration=4.867981073 podStartE2EDuration="4.867981073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:20.650958148 +0000 UTC m=+1330.772010972" watchObservedRunningTime="2026-11-03 16:34:20.867981073 +0000 UTC m=+1330.989033915" Nov 03 16:34:20.978531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.912655 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:20.978531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.912733 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:20.978531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:20.966662 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:21.374231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.318691 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:21.374231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.320992 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:21.374231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.321066 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:21.459116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.459058 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:21.459414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.459365 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:21.643303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.638536 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:21.700228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.691115 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:21.700597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.700560 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:21.814320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.814287 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:21.911276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.911165 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:21.911520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.910545 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:21.937235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.889358 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:21.957706 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.951414 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:22.063234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:21.967313 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.063234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.000303 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 16:34:22.063234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.012459 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.075848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.075802 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.120234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.102181 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.187281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.169911 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 16:34:22.207232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.201385 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:34:22.279462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.279416 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:22.279848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.279795 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:22.280174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.280119 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.332941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.332899 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": context deadline exceeded" Nov 03 16:34:22.561535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.525008 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.581273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.554513 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:22.581273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.562157 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:22.581273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.576422 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:22.581273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.562522 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:22.581273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.576496 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:22.607235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.605842 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.660297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.652490 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.660297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.652897 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:34:22.740524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.740481 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:22.805232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.804608 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:22.814689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.814599 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:22.814946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:22.814918 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:23.030249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.003859 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:34:23.030249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.010333 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:34:23.059248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.054051 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:34:23.059248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.054528 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:34:23.059248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.056681 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 16:34:23.059248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.057927 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:34:23.081341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.060098 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 16:34:23.081341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.060445 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:34:23.108651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.108605 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 16:34:23.109287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.109265 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:34:23.317287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.313173 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 16:34:23.317287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.313583 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:23.317287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.315529 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:34:23.317287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.315591 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:34:23.402274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.402166 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:23.404170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.404137 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:23.450961 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.450924 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:23.452354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.452321 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:23.608084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.608048 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:23.611753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.611712 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:23.615371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.615334 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:23.632373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.632342 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:34:23.652953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.650293 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 16:34:23.664238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.655881 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:34:23.674252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.665027 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 16:34:23.855232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.837305 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 16:34:23.855232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.837752 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:34:23.855232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.839455 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:23.855232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.839499 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:24.049290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:23.954274 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:34:24.465293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.018065 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:24.465293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.018115 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.097281 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.100169 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.100232 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.145902 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.280170 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.492471 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.316402 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 16:34:24.503420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.316356 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:24.565233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.468751 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:24.565233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.559320 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:24.565233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.559377 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:24.601348 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.601298 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:24.683457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.683418 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 16:34:24.715126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.689491 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:24.938234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.855637 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 16:34:24.992234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.945343 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:24.992234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.945412 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:24.992234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:24.964619 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.246291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.214761 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:34:25.296935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.296895 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:25.297177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.297148 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:25.473611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.471336 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.473611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.471888 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.473611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.472377 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:25.473611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.472422 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:25.473611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.472976 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.481252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.477324 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.481252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.477645 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:34:25.510395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.501807 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 16:34:25.510395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.502846 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.596233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.577354 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:25.596233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.577416 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:25.711556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.710389 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.711556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.710799 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:25.825158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.825062 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:25.852349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.852301 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:25.966384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:25.966333 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 16:34:26.228562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.094950 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:26.259232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.231401 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:26.366527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.366487 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:26.366701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.366565 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:26.391254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.378630 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Nov 03 16:34:26.450536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.450502 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:26.450802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.450772 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:26.533347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.486599 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:26.533347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.496568 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 16:34:26.533347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.496941 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:34:26.549021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.547383 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:26.549021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.547797 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 16:34:26.551768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.550787 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:26.551768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.551446 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 16:34:26.574237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.574026 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:26.574237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.574109 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:26.605134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.605101 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:26.746230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.740529 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 16:34:26.784357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.784247 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:26.867476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.867441 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:26.867751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.867722 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:26.870246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.870223 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:34:26.889272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.870732 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:34:26.889578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:26.889556 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 16:34:27.152796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.152698 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Nov 03 16:34:27.223609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.223386 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:27.271683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.245370 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 16:34:27.271683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.245751 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:34:27.281614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.281494 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:27.281614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.281556 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:27.311252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.294179 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:27.607573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.605923 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:27.607573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.605979 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:27.607573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.606058 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:27.607573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.606079 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:27.631292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.609277 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:34:27.631292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.609626 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 16:34:27.668234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.649363 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:27.668234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.649759 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 16:34:27.668234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.650068 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:34:27.851477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.851431 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 16:34:27.852947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.852922 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 16:34:27.896054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.891676 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:27.896054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.893571 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:27.896054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.893624 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:27.896054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.894473 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:34:27.896054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.895626 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 16:34:27.915232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.913733 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:34:27.935227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.929683 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:27.957234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.957176 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 16:34:27.966254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.965632 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 16:34:27.966663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.966646 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 16:34:27.973238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:27.972816 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:34:28.042818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.042787 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:34:28.088035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.087995 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:28.287669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.287631 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 16:34:28.289132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.289097 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 16:34:28.443243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.439625 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 16:34:28.443243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.439690 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 16:34:28.571543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.560979 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 16:34:28.571543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.561045 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 16:34:28.813786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.813746 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 16:34:28.814040 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:28.814004 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 16:34:29.561989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.560516 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:29.574639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.574460 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:29.575900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.575850 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:34:29.657464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.657421 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:29.658689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.658664 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:29.660511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.660482 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:34:29.898305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.897551 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:29.903101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.899405 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:29.903101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:29.900636 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:34:30.050445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:30.050325 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:30.051136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:30.051070 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:30.052148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:30.052094 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:34:31.815468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.815424 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8"] Nov 03 16:34:31.820474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.820449 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.911631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.911601 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8"] Nov 03 16:34:31.947302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947268 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947425 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqs2x\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-kube-api-access-sqs2x\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947497 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947554 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947619 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947597 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947724 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947776 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947839 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.947969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.947953 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:31.955640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:31.955615 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:32.049871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.049839 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050070 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050158 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqs2x\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-kube-api-access-sqs2x\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050188 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050238 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050284 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050238 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050325 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050405 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050412 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050457 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050468 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.050846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.050826 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.051123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.051096 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.052944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.052919 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.053419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.053399 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.110925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.110848 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqs2x\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-kube-api-access-sqs2x\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.111049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.111029 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.136299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.136270 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:32.587329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.587296 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8"] Nov 03 16:34:32.620372 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:34:32.620347 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb495a0_0a65_49d7_bb59_ee9f47188ff1.slice/crio-84dd9e37f6201854fc24642b93a16ea6658487408042607309ab0b426b4c5320 WatchSource:0}: Error finding container 84dd9e37f6201854fc24642b93a16ea6658487408042607309ab0b426b4c5320: Status 404 returned error can't find the container with id 84dd9e37f6201854fc24642b93a16ea6658487408042607309ab0b426b4c5320 Nov 03 16:34:32.623135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.623103 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:32.623273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.623172 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:32.623273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:32.623238 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:34:33.075030 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:33.074990 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" event={"ID":"3bb495a0-0a65-49d7-bb59-ee9f47188ff1","Type":"ContainerStarted","Data":"502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295"} Nov 03 16:34:33.075462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:33.075095 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" event={"ID":"3bb495a0-0a65-49d7-bb59-ee9f47188ff1","Type":"ContainerStarted","Data":"84dd9e37f6201854fc24642b93a16ea6658487408042607309ab0b426b4c5320"} Nov 03 16:34:33.140845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:33.140795 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:33.141033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:33.140953 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:34:33.141033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:33.141014 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 16:34:33.199487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:33.199438 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podStartSLOduration=2.199424189 podStartE2EDuration="2.199424189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:33.196748351 +0000 UTC m=+1343.317801174" watchObservedRunningTime="2026-11-03 16:34:33.199424189 +0000 UTC m=+1343.320477011" Nov 03 16:34:33.238264 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:33.229065 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:34.138537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:34.137654 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:34:34.138537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:34.137733 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 16:34:35.138752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:35.138703 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:34:35.139288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:35.138789 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 16:34:36.138714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:36.138681 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:34:36.138944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:36.138742 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 16:34:37.137685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:37.137643 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:34:37.138007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:37.137970 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 16:34:38.138629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:38.138590 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:34:38.139124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:38.138653 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 16:34:38.635226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:38.635158 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:39.136907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:39.136870 5946 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:34:39.137099 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:39.136933 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 16:34:40.141642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:40.141611 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:40.142119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:40.142105 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:40.142931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:40.142907 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:34:42.587838 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:34:42.587800 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:34:43.671050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:43.671017 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:48.796548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:48.796497 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:50.570860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:50.570827 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:53.927721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:53.927684 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:54.113958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:54.113929 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:54.320433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:54.320401 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:55.234774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:55.234709 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:55.418555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:55.418516 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:58.853359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:34:58.853326 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:03.677029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:03.676968 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:08.488299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:08.488246 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:12.587484 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:35:12.587447 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:35:13.197156 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:13.197105 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:17.863718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:17.863680 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:20.996596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:20.996565 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:35:23.207276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:23.207180 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:23.442855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:23.442817 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:26.075474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.075434 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9"] Nov 03 16:35:26.081651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.081618 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.085844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.085818 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9"] Nov 03 16:35:26.171233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.166091 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:35:26.234233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.217502 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:35:26.234233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.217588 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:35:26.234233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.217693 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:35:26.238231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.236122 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.238231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.236246 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.238231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.236362 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzbw\" (UniqueName: \"kubernetes.io/projected/020f3135-815f-45b8-a4e5-66d9a54e4366-kube-api-access-tnzbw\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.350394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.350300 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.350663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.350639 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.350847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.350831 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzbw\" (UniqueName: \"kubernetes.io/projected/020f3135-815f-45b8-a4e5-66d9a54e4366-kube-api-access-tnzbw\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.352055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.352028 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.366395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.366357 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.412703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.412670 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzbw\" (UniqueName: \"kubernetes.io/projected/020f3135-815f-45b8-a4e5-66d9a54e4366-kube-api-access-tnzbw\") pod \"maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:26.697606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:26.697519 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:27.349082 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:35:27.349055 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020f3135_815f_45b8_a4e5_66d9a54e4366.slice/crio-9c942cd4b970417e3f5a17cf5ecf9db3bb6579f23cd740307706ab6429610179 WatchSource:0}: Error finding container 9c942cd4b970417e3f5a17cf5ecf9db3bb6579f23cd740307706ab6429610179: Status 404 returned error can't find the container with id 9c942cd4b970417e3f5a17cf5ecf9db3bb6579f23cd740307706ab6429610179 Nov 03 16:35:27.353287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:27.353254 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9"] Nov 03 16:35:27.676774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:27.676692 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" event={"ID":"020f3135-815f-45b8-a4e5-66d9a54e4366","Type":"ContainerStarted","Data":"9c942cd4b970417e3f5a17cf5ecf9db3bb6579f23cd740307706ab6429610179"} Nov 03 16:35:28.690414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:28.690322 5946 generic.go:356] "Generic (PLEG): container finished" podID="020f3135-815f-45b8-a4e5-66d9a54e4366" containerID="dd3ee77e79232fc906354bd4aebd21f17b56f8e5ce44a068937ee5b18387aa50" exitCode=0 Nov 03 16:35:28.690414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:28.690388 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" event={"ID":"020f3135-815f-45b8-a4e5-66d9a54e4366","Type":"ContainerDied","Data":"dd3ee77e79232fc906354bd4aebd21f17b56f8e5ce44a068937ee5b18387aa50"} Nov 03 16:35:29.113698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.113605 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:29.902710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.902685 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:29.953944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.953865 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020f3135-815f-45b8-a4e5-66d9a54e4366-kube-api-access-tnzbw\" (UniqueName: \"kubernetes.io/projected/020f3135-815f-45b8-a4e5-66d9a54e4366-kube-api-access-tnzbw\") pod \"020f3135-815f-45b8-a4e5-66d9a54e4366\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " Nov 03 16:35:29.954096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.954027 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-xks-ca\") pod \"020f3135-815f-45b8-a4e5-66d9a54e4366\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " Nov 03 16:35:29.954096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.954066 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-openshift-ca\") pod \"020f3135-815f-45b8-a4e5-66d9a54e4366\" (UID: \"020f3135-815f-45b8-a4e5-66d9a54e4366\") " Nov 03 16:35:29.954469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.954438 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-openshift-ca" pod "020f3135-815f-45b8-a4e5-66d9a54e4366" (UID: "020f3135-815f-45b8-a4e5-66d9a54e4366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:35:29.955096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.955069 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:29.956292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.956268 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020f3135-815f-45b8-a4e5-66d9a54e4366-kube-api-access-tnzbw" pod "020f3135-815f-45b8-a4e5-66d9a54e4366" (UID: "020f3135-815f-45b8-a4e5-66d9a54e4366"). InnerVolumeSpecName "kube-api-access-tnzbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:35:29.956394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:29.956352 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-xks-ca" pod "020f3135-815f-45b8-a4e5-66d9a54e4366" (UID: "020f3135-815f-45b8-a4e5-66d9a54e4366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:35:30.057660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:30.057624 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnzbw\" (UniqueName: \"kubernetes.io/projected/020f3135-815f-45b8-a4e5-66d9a54e4366-kube-api-access-tnzbw\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:30.057660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:30.057660 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020f3135-815f-45b8-a4e5-66d9a54e4366-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:30.715959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:30.715929 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" Nov 03 16:35:30.715959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:30.715958 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9" event={"ID":"020f3135-815f-45b8-a4e5-66d9a54e4366","Type":"ContainerDied","Data":"9c942cd4b970417e3f5a17cf5ecf9db3bb6579f23cd740307706ab6429610179"} Nov 03 16:35:30.716268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:30.716009 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c942cd4b970417e3f5a17cf5ecf9db3bb6579f23cd740307706ab6429610179" Nov 03 16:35:32.328519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.328484 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:32.744932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.744901 5946 generic.go:356] "Generic (PLEG): container finished" podID="f24c959b-1822-45e4-a423-942e8af1fb26" containerID="d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be" exitCode=0 Nov 03 16:35:32.745130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.744984 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" event={"ID":"f24c959b-1822-45e4-a423-942e8af1fb26","Type":"ContainerDied","Data":"d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be"} Nov 03 16:35:32.745530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.745508 5946 scope.go:122] "RemoveContainer" containerID="d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be" Nov 03 16:35:32.746859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.746835 5946 generic.go:356] "Generic (PLEG): container finished" podID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerID="09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d" exitCode=0 Nov 03 16:35:32.746992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.746902 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" event={"ID":"3d12746a-0441-43a7-b3a0-18ce49c446ea","Type":"ContainerDied","Data":"09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d"} Nov 03 16:35:32.747302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.747277 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:32.747369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.747355 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:32.747406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.747369 5946 scope.go:122] "RemoveContainer" containerID="09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d" Nov 03 16:35:32.747406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.747400 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:32.748891 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.748868 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:32.748963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.748930 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:32.749001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:32.748972 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:33.773331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:33.773296 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" event={"ID":"f24c959b-1822-45e4-a423-942e8af1fb26","Type":"ContainerStarted","Data":"0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781"} Nov 03 16:35:33.775702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:33.775665 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" event={"ID":"3d12746a-0441-43a7-b3a0-18ce49c446ea","Type":"ContainerStarted","Data":"fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6"} Nov 03 16:35:34.198103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:34.198024 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:35:34.198800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:34.198774 5946 patch_prober.go:29] interesting pod/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 16:35:34.198867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:34.198824 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 16:35:34.337718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:34.337686 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:34.735649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:34.735616 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:35:34.736022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:34.735998 5946 patch_prober.go:29] interesting pod/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:35:34.736195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:34.736168 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 16:35:35.198172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:35.198069 5946 patch_prober.go:29] interesting pod/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 16:35:35.198172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:35.198140 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 16:35:35.740231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:35.734615 5946 patch_prober.go:29] interesting pod/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:35:35.740231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:35.734696 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 16:35:35.756367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:35.756323 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:35:36.197819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:36.197725 5946 patch_prober.go:29] interesting pod/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 16:35:36.197819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:36.197796 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 16:35:36.493768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:36.493739 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl"] Nov 03 16:35:36.740620 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:36.740584 5946 patch_prober.go:29] interesting pod/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:35:36.740893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:36.740864 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 16:35:36.759406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:36.759310 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:35:37.000235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:36.998589 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:37.200317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:37.198901 5946 patch_prober.go:29] interesting pod/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 16:35:37.200317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:37.198975 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 16:35:37.293389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:37.293353 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:37.736697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:37.736655 5946 patch_prober.go:29] interesting pod/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:35:37.746234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:37.737317 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 16:35:38.201673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:38.201579 5946 patch_prober.go:29] interesting pod/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 16:35:38.203058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:38.203022 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 16:35:38.737831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:38.737795 5946 patch_prober.go:29] interesting pod/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:35:38.738450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:38.738417 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 16:35:39.199632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:39.199062 5946 patch_prober.go:29] interesting pod/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 16:35:39.199632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:39.199125 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 16:35:39.220852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:39.220605 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:39.501056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:39.501026 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:39.738870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:39.738840 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:35:39.739983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:39.739954 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:35:39.961345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:39.961238 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:40.201753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:40.201720 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:35:40.202937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:40.202920 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:35:40.376823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:40.376787 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:41.221716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.221668 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl"] Nov 03 16:35:41.222446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.222426 5946 cpu_manager.go:414] "containerMap: removing container" podUID="020f3135-815f-45b8-a4e5-66d9a54e4366" containerName="revoke-keys" Nov 03 16:35:41.222446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.222445 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020f3135-815f-45b8-a4e5-66d9a54e4366" containerName="revoke-keys" Nov 03 16:35:41.222597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.222577 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="020f3135-815f-45b8-a4e5-66d9a54e4366" containerName="revoke-keys" Nov 03 16:35:41.225470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.225438 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.244817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.244767 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523fd694-634d-4bd0-b573-773d9e42feeb-istiod-ca-cert\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.244996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.244837 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6p4\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-kube-api-access-6c6p4\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.245087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.245005 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-socket\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.245148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.245118 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-credential-socket\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.245148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.245138 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-certs\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.245281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.245171 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-envoy\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.245281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.245247 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523fd694-634d-4bd0-b573-773d9e42feeb-istio-podinfo\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.245391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.245278 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-istio-token\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.245391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.245314 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-data\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.329915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.329879 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl"] Nov 03 16:35:41.345992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.345959 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-9f2868-openshift-default-dockercfg-l2qcx\"" Nov 03 16:35:41.346399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346375 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-socket\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346502 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-credential-socket\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346532 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-certs\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346576 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-envoy\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346636 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523fd694-634d-4bd0-b573-773d9e42feeb-istio-podinfo\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346668 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-istio-token\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346703 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-data\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346842 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-socket\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.346898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346852 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523fd694-634d-4bd0-b573-773d9e42feeb-istiod-ca-cert\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.347035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346921 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6p4\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-kube-api-access-6c6p4\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.347133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.346842 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-credential-socket\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.347876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.347566 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-certs\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.347876 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.347581 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523fd694-634d-4bd0-b573-773d9e42feeb-istiod-ca-cert\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.348079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.347819 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-data\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.350047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.350022 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-envoy\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.350221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.350189 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523fd694-634d-4bd0-b573-773d9e42feeb-istio-podinfo\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.390426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.390386 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:41.417282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.417245 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-istio-token\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.423662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.423560 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6p4\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-kube-api-access-6c6p4\") pod \"e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.497234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.494195 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" podUID="1c69528a-235b-4f9b-a98b-7c8901709517" containerName="payload-processing" containerID="cri-o://a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2" gracePeriod=25 Nov 03 16:35:41.541649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.541611 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:41.718257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.718165 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:41.856424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.856398 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:35:41.895581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.895547 5946 generic.go:356] "Generic (PLEG): container finished" podID="1c69528a-235b-4f9b-a98b-7c8901709517" containerID="a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2" exitCode=0 Nov 03 16:35:41.895753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.895601 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" Nov 03 16:35:41.895753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.895637 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" event={"ID":"1c69528a-235b-4f9b-a98b-7c8901709517","Type":"ContainerDied","Data":"a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2"} Nov 03 16:35:41.895753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.895701 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl" event={"ID":"1c69528a-235b-4f9b-a98b-7c8901709517","Type":"ContainerDied","Data":"a189b03f322eb1413bbaca679a098156da2e8993fe07720c4a1fc8db3a9d7f7f"} Nov 03 16:35:41.895753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.895733 5946 scope.go:122] "RemoveContainer" containerID="a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2" Nov 03 16:35:41.907795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.907771 5946 scope.go:122] "RemoveContainer" containerID="a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2" Nov 03 16:35:41.908109 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:35:41.908088 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2\": container with ID starting with a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2 not found: ID does not exist" containerID="a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2" Nov 03 16:35:41.908168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.908121 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2"} err="failed to get container status \"a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2\": rpc error: code = NotFound desc = could not find container \"a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2\": container with ID starting with a4bc6420881a6f7b23859a8dda9a31810869d67897b0afe51dd6e4ff86069df2 not found: ID does not exist" Nov 03 16:35:41.967558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.967521 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c69528a-235b-4f9b-a98b-7c8901709517-kube-api-access-252s7\" (UniqueName: \"kubernetes.io/projected/1c69528a-235b-4f9b-a98b-7c8901709517-kube-api-access-252s7\") pod \"1c69528a-235b-4f9b-a98b-7c8901709517\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " Nov 03 16:35:41.967767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.967749 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c69528a-235b-4f9b-a98b-7c8901709517-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69528a-235b-4f9b-a98b-7c8901709517-plugins-config-volume\") pod \"1c69528a-235b-4f9b-a98b-7c8901709517\" (UID: \"1c69528a-235b-4f9b-a98b-7c8901709517\") " Nov 03 16:35:41.968109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.968081 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c69528a-235b-4f9b-a98b-7c8901709517-plugins-config-volume" pod "1c69528a-235b-4f9b-a98b-7c8901709517" (UID: "1c69528a-235b-4f9b-a98b-7c8901709517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:35:41.968508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.968490 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69528a-235b-4f9b-a98b-7c8901709517-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:41.969845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:41.969825 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c69528a-235b-4f9b-a98b-7c8901709517-kube-api-access-252s7" pod "1c69528a-235b-4f9b-a98b-7c8901709517" (UID: "1c69528a-235b-4f9b-a98b-7c8901709517"). InnerVolumeSpecName "kube-api-access-252s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:35:42.032514 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:35:42.032477 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523fd694_634d_4bd0_b573_773d9e42feeb.slice/crio-5f38918fbe858781c462f94158a12e6bd859d4efadb209261943a0a758de44a4 WatchSource:0}: Error finding container 5f38918fbe858781c462f94158a12e6bd859d4efadb209261943a0a758de44a4: Status 404 returned error can't find the container with id 5f38918fbe858781c462f94158a12e6bd859d4efadb209261943a0a758de44a4 Nov 03 16:35:42.036296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.036259 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:42.036474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.036452 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl"] Nov 03 16:35:42.036578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.036555 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:42.036677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.036615 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:42.070595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.070562 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252s7\" (UniqueName: \"kubernetes.io/projected/1c69528a-235b-4f9b-a98b-7c8901709517-kube-api-access-252s7\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:42.101682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.101646 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4"] Nov 03 16:35:42.102350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.102330 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1c69528a-235b-4f9b-a98b-7c8901709517" containerName="payload-processing" Nov 03 16:35:42.102350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.102347 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c69528a-235b-4f9b-a98b-7c8901709517" containerName="payload-processing" Nov 03 16:35:42.102524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.102481 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c69528a-235b-4f9b-a98b-7c8901709517" containerName="payload-processing" Nov 03 16:35:42.106047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.106026 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.171374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.171343 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:35:42.171573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.171433 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:35:42.172124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.172093 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgm5\" (UniqueName: \"kubernetes.io/projected/924c3616-7905-483f-b43d-3859858a80b9-kube-api-access-dhgm5\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.172365 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.172346 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924c3616-7905-483f-b43d-3859858a80b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.172499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.172483 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924c3616-7905-483f-b43d-3859858a80b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.178535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.178274 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:35:42.178535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.178340 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:35:42.182873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.182250 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4"] Nov 03 16:35:42.274163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.274120 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924c3616-7905-483f-b43d-3859858a80b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.274707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.274264 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924c3616-7905-483f-b43d-3859858a80b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.274707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.274330 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgm5\" (UniqueName: \"kubernetes.io/projected/924c3616-7905-483f-b43d-3859858a80b9-kube-api-access-dhgm5\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.274990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.274967 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924c3616-7905-483f-b43d-3859858a80b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.276831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.276810 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924c3616-7905-483f-b43d-3859858a80b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.327418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.327383 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:42.353294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.353257 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgm5\" (UniqueName: \"kubernetes.io/projected/924c3616-7905-483f-b43d-3859858a80b9-kube-api-access-dhgm5\") pod \"maas-api-revoke-keys-e2e-ait-1b774106-kr6c4\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.377054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.377014 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl"] Nov 03 16:35:42.388903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.388868 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e998bdb4-65bbf94645-6vrgl"] Nov 03 16:35:42.422463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.422433 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:42.624444 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:35:42.624344 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:35:42.753042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.753011 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:42.847488 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:35:42.847459 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924c3616_7905_483f_b43d_3859858a80b9.slice/crio-7f3dff99a4669253f6500f3fff6df3522000e495dd33a23fccadf159e643bcb3 WatchSource:0}: Error finding container 7f3dff99a4669253f6500f3fff6df3522000e495dd33a23fccadf159e643bcb3: Status 404 returned error can't find the container with id 7f3dff99a4669253f6500f3fff6df3522000e495dd33a23fccadf159e643bcb3 Nov 03 16:35:42.847853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.847831 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4"] Nov 03 16:35:42.906270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.906238 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" event={"ID":"523fd694-634d-4bd0-b573-773d9e42feeb","Type":"ContainerStarted","Data":"7c1123317bc485eb77f316f2bd3c225dc026166e13a7605f2b48f96b66f0d053"} Nov 03 16:35:42.906410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.906291 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" event={"ID":"523fd694-634d-4bd0-b573-773d9e42feeb","Type":"ContainerStarted","Data":"5f38918fbe858781c462f94158a12e6bd859d4efadb209261943a0a758de44a4"} Nov 03 16:35:42.908632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:42.908610 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" event={"ID":"924c3616-7905-483f-b43d-3859858a80b9","Type":"ContainerStarted","Data":"7f3dff99a4669253f6500f3fff6df3522000e495dd33a23fccadf159e643bcb3"} Nov 03 16:35:43.030445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:43.030392 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podStartSLOduration=2.030377387 podStartE2EDuration="2.030377387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:43.029870283 +0000 UTC m=+1413.150923107" watchObservedRunningTime="2026-11-03 16:35:43.030377387 +0000 UTC m=+1413.151430210" Nov 03 16:35:43.541903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:43.541872 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:43.547617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:43.547588 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:43.921814 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:43.921776 5946 generic.go:356] "Generic (PLEG): container finished" podID="924c3616-7905-483f-b43d-3859858a80b9" containerID="56d124e6382ca58f447d49b1d9e22ba16a3d16516da471a2bfe882e534d32fd4" exitCode=0 Nov 03 16:35:43.921979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:43.921863 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" event={"ID":"924c3616-7905-483f-b43d-3859858a80b9","Type":"ContainerDied","Data":"56d124e6382ca58f447d49b1d9e22ba16a3d16516da471a2bfe882e534d32fd4"} Nov 03 16:35:43.922277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:43.922260 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:43.923122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:43.923103 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:35:44.006554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:44.006523 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c69528a-235b-4f9b-a98b-7c8901709517" path="/var/lib/kubelet/pods/1c69528a-235b-4f9b-a98b-7c8901709517/volumes" Nov 03 16:35:45.182232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.175378 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:45.235357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.235320 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924c3616-7905-483f-b43d-3859858a80b9-kube-api-access-dhgm5\" (UniqueName: \"kubernetes.io/projected/924c3616-7905-483f-b43d-3859858a80b9-kube-api-access-dhgm5\") pod \"924c3616-7905-483f-b43d-3859858a80b9\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " Nov 03 16:35:45.235533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.235431 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924c3616-7905-483f-b43d-3859858a80b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924c3616-7905-483f-b43d-3859858a80b9-maas-api-openshift-ca\") pod \"924c3616-7905-483f-b43d-3859858a80b9\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " Nov 03 16:35:45.235593 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.235559 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/924c3616-7905-483f-b43d-3859858a80b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924c3616-7905-483f-b43d-3859858a80b9-maas-api-xks-ca\") pod \"924c3616-7905-483f-b43d-3859858a80b9\" (UID: \"924c3616-7905-483f-b43d-3859858a80b9\") " Nov 03 16:35:45.236919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.236858 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924c3616-7905-483f-b43d-3859858a80b9-maas-api-openshift-ca" pod "924c3616-7905-483f-b43d-3859858a80b9" (UID: "924c3616-7905-483f-b43d-3859858a80b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:35:45.238575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.238550 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/924c3616-7905-483f-b43d-3859858a80b9-maas-api-xks-ca" pod "924c3616-7905-483f-b43d-3859858a80b9" (UID: "924c3616-7905-483f-b43d-3859858a80b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:35:45.239346 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.239326 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924c3616-7905-483f-b43d-3859858a80b9-kube-api-access-dhgm5" pod "924c3616-7905-483f-b43d-3859858a80b9" (UID: "924c3616-7905-483f-b43d-3859858a80b9"). InnerVolumeSpecName "kube-api-access-dhgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:35:45.342229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.338512 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924c3616-7905-483f-b43d-3859858a80b9-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:45.342229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.338551 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924c3616-7905-483f-b43d-3859858a80b9-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:45.342229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.338567 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhgm5\" (UniqueName: \"kubernetes.io/projected/924c3616-7905-483f-b43d-3859858a80b9-kube-api-access-dhgm5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:35:45.553239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.553133 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:45.943136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.943053 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" event={"ID":"924c3616-7905-483f-b43d-3859858a80b9","Type":"ContainerDied","Data":"7f3dff99a4669253f6500f3fff6df3522000e495dd33a23fccadf159e643bcb3"} Nov 03 16:35:45.943308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.943146 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3dff99a4669253f6500f3fff6df3522000e495dd33a23fccadf159e643bcb3" Nov 03 16:35:45.943308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:45.943068 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4" Nov 03 16:35:46.103998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:46.103972 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:46.435387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:46.435356 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:49.506250 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:49.504783 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:49.781557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:49.781459 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:50.048293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.047233 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks"] Nov 03 16:35:50.048293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.048257 5946 cpu_manager.go:414] "containerMap: removing container" podUID="924c3616-7905-483f-b43d-3859858a80b9" containerName="revoke-keys" Nov 03 16:35:50.048293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.048280 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924c3616-7905-483f-b43d-3859858a80b9" containerName="revoke-keys" Nov 03 16:35:50.057911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.048611 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="924c3616-7905-483f-b43d-3859858a80b9" containerName="revoke-keys" Nov 03 16:35:50.069628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.069603 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.075266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.074465 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks"] Nov 03 16:35:50.146889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.146853 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9f2868\"" Nov 03 16:35:50.216233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.216040 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9f2868-dockercfg-lhvrh\"" Nov 03 16:35:50.408165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.375539 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqp66\" (UniqueName: \"kubernetes.io/projected/c7497b27-415b-4819-93df-ef43132f4f85-kube-api-access-qqp66\") pod \"payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.438639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.438606 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7497b27-415b-4819-93df-ef43132f4f85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.451334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.451297 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds"] Nov 03 16:35:50.455426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.455403 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.457496 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.457466 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds"] Nov 03 16:35:50.541231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.541031 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqp66\" (UniqueName: \"kubernetes.io/projected/c7497b27-415b-4819-93df-ef43132f4f85-kube-api-access-qqp66\") pod \"payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.541231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.541127 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpxb\" (UniqueName: \"kubernetes.io/projected/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-kube-api-access-fgpxb\") pod \"payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.541231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.541172 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7497b27-415b-4819-93df-ef43132f4f85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.541942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.541248 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.542768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.542741 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7497b27-415b-4819-93df-ef43132f4f85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.603140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.603108 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqp66\" (UniqueName: \"kubernetes.io/projected/c7497b27-415b-4819-93df-ef43132f4f85-kube-api-access-qqp66\") pod \"payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.643585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.643520 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.644043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.644019 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpxb\" (UniqueName: \"kubernetes.io/projected/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-kube-api-access-fgpxb\") pod \"payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.644479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.644458 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.688286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.688188 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:50.707890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.707867 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpxb\" (UniqueName: \"kubernetes.io/projected/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-kube-api-access-fgpxb\") pod \"payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.772893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.772858 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:50.991087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:50.991057 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:51.142622 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:35:51.142588 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7497b27_415b_4819_93df_ef43132f4f85.slice/crio-187d05eff792186aab1a4126286e6f5655704e86ab7f4434dbd45c5b5a9397db WatchSource:0}: Error finding container 187d05eff792186aab1a4126286e6f5655704e86ab7f4434dbd45c5b5a9397db: Status 404 returned error can't find the container with id 187d05eff792186aab1a4126286e6f5655704e86ab7f4434dbd45c5b5a9397db Nov 03 16:35:51.144637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:51.144615 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks"] Nov 03 16:35:51.245484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:51.245394 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds"] Nov 03 16:35:51.252175 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:35:51.252138 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580a4eeb_efa9_490f_bb83_8ca3dc7e95a3.slice/crio-79168c25e91ab81bb7a8f2c5a7ffbf576f981d4f20470d290591e723edba5d74 WatchSource:0}: Error finding container 79168c25e91ab81bb7a8f2c5a7ffbf576f981d4f20470d290591e723edba5d74: Status 404 returned error can't find the container with id 79168c25e91ab81bb7a8f2c5a7ffbf576f981d4f20470d290591e723edba5d74 Nov 03 16:35:51.383086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:51.383049 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:52.012286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.012250 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" event={"ID":"c7497b27-415b-4819-93df-ef43132f4f85","Type":"ContainerStarted","Data":"3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a"} Nov 03 16:35:52.012731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.012323 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" event={"ID":"c7497b27-415b-4819-93df-ef43132f4f85","Type":"ContainerStarted","Data":"187d05eff792186aab1a4126286e6f5655704e86ab7f4434dbd45c5b5a9397db"} Nov 03 16:35:52.012731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.012396 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:35:52.040231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.040083 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" event={"ID":"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3","Type":"ContainerStarted","Data":"821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d"} Nov 03 16:35:52.040231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.040131 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" event={"ID":"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3","Type":"ContainerStarted","Data":"79168c25e91ab81bb7a8f2c5a7ffbf576f981d4f20470d290591e723edba5d74"} Nov 03 16:35:52.043235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.040739 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:35:52.146973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.146915 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" podStartSLOduration=3.146900361 podStartE2EDuration="3.146900361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:52.142633138 +0000 UTC m=+1422.263685961" watchObservedRunningTime="2026-11-03 16:35:52.146900361 +0000 UTC m=+1422.267953183" Nov 03 16:35:52.269793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:52.269694 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" podStartSLOduration=2.269677923 podStartE2EDuration="2.269677923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:52.268381972 +0000 UTC m=+1422.389434797" watchObservedRunningTime="2026-11-03 16:35:52.269677923 +0000 UTC m=+1422.390730746" Nov 03 16:35:53.086998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:53.086962 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z"] Nov 03 16:35:53.139932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:53.139899 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e01dbe-t545z"] Nov 03 16:35:53.328701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:53.328666 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:53.719144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:53.718870 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:54.002615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:54.002589 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:54.007437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:54.007412 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f998d91-333b-43fc-a7a2-79ff8a974c60" path="/var/lib/kubelet/pods/9f998d91-333b-43fc-a7a2-79ff8a974c60/volumes" Nov 03 16:35:56.909384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:56.909354 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:57.109928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:57.109847 5946 generic.go:356] "Generic (PLEG): container finished" podID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerID="6a019ba89d568f8299b1030aa2adc8349f53f36b4f8f0b87b8d5c69b677e2c98" exitCode=0 Nov 03 16:35:57.110066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:57.109930 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" event={"ID":"f8e5925d-2e74-4a38-a86b-5bbb45424268","Type":"ContainerDied","Data":"6a019ba89d568f8299b1030aa2adc8349f53f36b4f8f0b87b8d5c69b677e2c98"} Nov 03 16:35:57.110410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:57.110395 5946 scope.go:122] "RemoveContainer" containerID="6a019ba89d568f8299b1030aa2adc8349f53f36b4f8f0b87b8d5c69b677e2c98" Nov 03 16:35:57.112048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:57.112017 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:57.112161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:57.112106 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:57.112161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:57.112154 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:35:58.141230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:58.139761 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" event={"ID":"f8e5925d-2e74-4a38-a86b-5bbb45424268","Type":"ContainerStarted","Data":"728dd2b5a6be80e249bb0eda2e62709bcb1f65a3ff6236caaeb22340bae7b41a"} Nov 03 16:35:58.242599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:58.242538 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:58.812349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:58.812315 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:35:58.812570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:58.812449 5946 patch_prober.go:29] interesting pod/e2e-derive-161f6056-openshift-default-c456fd685-trmm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 16:35:58.812570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:58.812505 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 16:35:59.812840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:59.812805 5946 patch_prober.go:29] interesting pod/e2e-derive-161f6056-openshift-default-c456fd685-trmm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 16:35:59.813281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:35:59.812855 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 16:36:00.813144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:00.813114 5946 patch_prober.go:29] interesting pod/e2e-derive-161f6056-openshift-default-c456fd685-trmm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 16:36:00.813678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:00.813161 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 16:36:01.813038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:01.813001 5946 patch_prober.go:29] interesting pod/e2e-derive-161f6056-openshift-default-c456fd685-trmm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 16:36:01.813282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:01.813073 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 16:36:02.559193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:02.555194 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:02.812228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:02.812125 5946 patch_prober.go:29] interesting pod/e2e-derive-161f6056-openshift-default-c456fd685-trmm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 16:36:02.812228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:02.812180 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 16:36:03.052489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:03.052455 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 16:36:03.832971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:03.832932 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:36:03.833406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:03.833383 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:36:03.834119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:03.834100 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:36:04.061386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:04.061356 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 16:36:05.073702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:05.073671 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm"] Nov 03 16:36:05.123962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:05.123931 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369ef4ba-qzsgm"] Nov 03 16:36:05.788633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:05.788588 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:06.006911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:06.006880 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34627a78-1e05-4fa1-9b89-ea2b386bc999" path="/var/lib/kubelet/pods/34627a78-1e05-4fa1-9b89-ea2b386bc999/volumes" Nov 03 16:36:06.735309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:06.735272 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:06.946718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:06.946686 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:07.403763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:07.403728 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:07.599168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:07.599138 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:12.164678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.164639 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt"] Nov 03 16:36:12.169247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.169194 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.229773 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.229749 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt"] Nov 03 16:36:12.241972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.241942 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.242252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.242236 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h624f\" (UniqueName: \"kubernetes.io/projected/8856be18-f4ac-4430-91f1-b0ba2283c09b-kube-api-access-h624f\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.242352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.242326 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.275398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.275361 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:36:12.275527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.275425 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:36:12.275527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.275447 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:36:12.275628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.275564 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:36:12.344018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.343990 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h624f\" (UniqueName: \"kubernetes.io/projected/8856be18-f4ac-4430-91f1-b0ba2283c09b-kube-api-access-h624f\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.344197 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.344024 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.344197 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.344163 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.344794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.344777 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.346491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.346471 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.405093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.405064 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h624f\" (UniqueName: \"kubernetes.io/projected/8856be18-f4ac-4430-91f1-b0ba2283c09b-kube-api-access-h624f\") pod \"maas-api-revoke-keys-e2e-migrate-e6def865-zspjt\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.482049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.482019 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:12.601446 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:36:12.601401 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:36:12.869921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.869769 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:12.880405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:12.880373 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt"] Nov 03 16:36:12.886913 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:36:12.886875 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8856be18_f4ac_4430_91f1_b0ba2283c09b.slice/crio-7f56078940bcdbafcd0299d31fba6529abff10072913b4775806c85ca0a4547b WatchSource:0}: Error finding container 7f56078940bcdbafcd0299d31fba6529abff10072913b4775806c85ca0a4547b: Status 404 returned error can't find the container with id 7f56078940bcdbafcd0299d31fba6529abff10072913b4775806c85ca0a4547b Nov 03 16:36:13.271502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:13.271467 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" event={"ID":"8856be18-f4ac-4430-91f1-b0ba2283c09b","Type":"ContainerStarted","Data":"c6cb2b4cc722635f0af460e70673f3fb834bc748cbc56f00f5f80d51e2125803"} Nov 03 16:36:13.271941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:13.271531 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" event={"ID":"8856be18-f4ac-4430-91f1-b0ba2283c09b","Type":"ContainerStarted","Data":"7f56078940bcdbafcd0299d31fba6529abff10072913b4775806c85ca0a4547b"} Nov 03 16:36:13.399894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:13.399799 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" podStartSLOduration=1.399787148 podStartE2EDuration="1.399787148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:13.395286768 +0000 UTC m=+1443.516339592" watchObservedRunningTime="2026-11-03 16:36:13.399787148 +0000 UTC m=+1443.520839971" Nov 03 16:36:14.282186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:14.282148 5946 generic.go:356] "Generic (PLEG): container finished" podID="8856be18-f4ac-4430-91f1-b0ba2283c09b" containerID="c6cb2b4cc722635f0af460e70673f3fb834bc748cbc56f00f5f80d51e2125803" exitCode=0 Nov 03 16:36:14.282716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:14.282240 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" event={"ID":"8856be18-f4ac-4430-91f1-b0ba2283c09b","Type":"ContainerDied","Data":"c6cb2b4cc722635f0af460e70673f3fb834bc748cbc56f00f5f80d51e2125803"} Nov 03 16:36:15.244933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.244904 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:15.464630 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.464599 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:15.598187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.598105 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8856be18-f4ac-4430-91f1-b0ba2283c09b-kube-api-access-h624f\" (UniqueName: \"kubernetes.io/projected/8856be18-f4ac-4430-91f1-b0ba2283c09b-kube-api-access-h624f\") pod \"8856be18-f4ac-4430-91f1-b0ba2283c09b\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " Nov 03 16:36:15.598412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.598287 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-openshift-ca\") pod \"8856be18-f4ac-4430-91f1-b0ba2283c09b\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " Nov 03 16:36:15.598412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.598358 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-xks-ca\") pod \"8856be18-f4ac-4430-91f1-b0ba2283c09b\" (UID: \"8856be18-f4ac-4430-91f1-b0ba2283c09b\") " Nov 03 16:36:15.598653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.598625 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-openshift-ca" pod "8856be18-f4ac-4430-91f1-b0ba2283c09b" (UID: "8856be18-f4ac-4430-91f1-b0ba2283c09b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:36:15.599184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.599163 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:15.600996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.600966 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-xks-ca" pod "8856be18-f4ac-4430-91f1-b0ba2283c09b" (UID: "8856be18-f4ac-4430-91f1-b0ba2283c09b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:36:15.601198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.601173 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8856be18-f4ac-4430-91f1-b0ba2283c09b-kube-api-access-h624f" pod "8856be18-f4ac-4430-91f1-b0ba2283c09b" (UID: "8856be18-f4ac-4430-91f1-b0ba2283c09b"). InnerVolumeSpecName "kube-api-access-h624f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:36:15.700457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.700423 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856be18-f4ac-4430-91f1-b0ba2283c09b-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:15.700457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:15.700453 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h624f\" (UniqueName: \"kubernetes.io/projected/8856be18-f4ac-4430-91f1-b0ba2283c09b-kube-api-access-h624f\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:16.301326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:16.301240 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" event={"ID":"8856be18-f4ac-4430-91f1-b0ba2283c09b","Type":"ContainerDied","Data":"7f56078940bcdbafcd0299d31fba6529abff10072913b4775806c85ca0a4547b"} Nov 03 16:36:16.301326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:16.301282 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt" Nov 03 16:36:16.301326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:16.301295 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f56078940bcdbafcd0299d31fba6529abff10072913b4775806c85ca0a4547b" Nov 03 16:36:18.086505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:18.086471 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:21.730240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:21.728320 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:36:22.869784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:22.869754 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:23.065728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:23.065693 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn"] Nov 03 16:36:23.116986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:23.116955 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738b2dd0-cl8mn"] Nov 03 16:36:24.015484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:24.015452 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f216f28-08a0-4051-b711-0c8f92f3ee14" path="/var/lib/kubelet/pods/4f216f28-08a0-4051-b711-0c8f92f3ee14/volumes" Nov 03 16:36:24.184601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:24.184555 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:24.400975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:24.400894 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:25.102555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.102522 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp"] Nov 03 16:36:25.103163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.103150 5946 cpu_manager.go:414] "containerMap: removing container" podUID="8856be18-f4ac-4430-91f1-b0ba2283c09b" containerName="revoke-keys" Nov 03 16:36:25.103245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.103164 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8856be18-f4ac-4430-91f1-b0ba2283c09b" containerName="revoke-keys" Nov 03 16:36:25.103392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.103380 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="8856be18-f4ac-4430-91f1-b0ba2283c09b" containerName="revoke-keys" Nov 03 16:36:25.106598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.106581 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.169936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.169904 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp"] Nov 03 16:36:25.213845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.213812 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:36:25.213845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.213834 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:36:25.213845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.213837 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:36:25.214109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.213927 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:36:25.258186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.258150 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgws\" (UniqueName: \"kubernetes.io/projected/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-kube-api-access-5lgws\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.258374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.258337 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.258478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.258462 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.363277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.360196 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.363277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.360432 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgws\" (UniqueName: \"kubernetes.io/projected/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-kube-api-access-5lgws\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.363277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.360563 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.363277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.361535 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.400826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.400795 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.452228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.447917 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgws\" (UniqueName: \"kubernetes.io/projected/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-kube-api-access-5lgws\") pod \"maas-api-revoke-keys-e2e-derive-161f6056-r4bcp\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:25.720151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:25.720027 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:26.144861 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:36:26.144831 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a24b991_0fff_4c98_8cdf_bbcfee1c261e.slice/crio-81bcd959bac3aa40ae898e2454ddc68b47e3842acc71398e4ad9e84f0e414c79 WatchSource:0}: Error finding container 81bcd959bac3aa40ae898e2454ddc68b47e3842acc71398e4ad9e84f0e414c79: Status 404 returned error can't find the container with id 81bcd959bac3aa40ae898e2454ddc68b47e3842acc71398e4ad9e84f0e414c79 Nov 03 16:36:26.146358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:26.146333 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp"] Nov 03 16:36:26.409037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:26.408953 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" event={"ID":"9a24b991-0fff-4c98-8cdf-bbcfee1c261e","Type":"ContainerStarted","Data":"81bcd959bac3aa40ae898e2454ddc68b47e3842acc71398e4ad9e84f0e414c79"} Nov 03 16:36:27.402015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:27.401936 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:27.420563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:27.420529 5946 generic.go:356] "Generic (PLEG): container finished" podID="9a24b991-0fff-4c98-8cdf-bbcfee1c261e" containerID="df5a5b0d8b832b44df97c24cfb714e70a80129dad56b48a5cebafeca9d5fa800" exitCode=0 Nov 03 16:36:27.420769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:27.420626 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" event={"ID":"9a24b991-0fff-4c98-8cdf-bbcfee1c261e","Type":"ContainerDied","Data":"df5a5b0d8b832b44df97c24cfb714e70a80129dad56b48a5cebafeca9d5fa800"} Nov 03 16:36:28.627061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.627038 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:28.739839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.739811 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-xks-ca\") pod \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " Nov 03 16:36:28.740045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.740012 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-kube-api-access-5lgws\" (UniqueName: \"kubernetes.io/projected/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-kube-api-access-5lgws\") pod \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " Nov 03 16:36:28.740177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.740157 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-openshift-ca\") pod \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\" (UID: \"9a24b991-0fff-4c98-8cdf-bbcfee1c261e\") " Nov 03 16:36:28.740515 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.740492 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-openshift-ca" pod "9a24b991-0fff-4c98-8cdf-bbcfee1c261e" (UID: "9a24b991-0fff-4c98-8cdf-bbcfee1c261e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:36:28.740864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.740843 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:28.742076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.742057 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-xks-ca" pod "9a24b991-0fff-4c98-8cdf-bbcfee1c261e" (UID: "9a24b991-0fff-4c98-8cdf-bbcfee1c261e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:36:28.742396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.742353 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-kube-api-access-5lgws" pod "9a24b991-0fff-4c98-8cdf-bbcfee1c261e" (UID: "9a24b991-0fff-4c98-8cdf-bbcfee1c261e"). InnerVolumeSpecName "kube-api-access-5lgws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:36:28.842818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.842768 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:28.842818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.842815 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgws\" (UniqueName: \"kubernetes.io/projected/9a24b991-0fff-4c98-8cdf-bbcfee1c261e-kube-api-access-5lgws\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:28.929763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:28.929731 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:29.227877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.227782 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:29.431386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.431352 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9"] Nov 03 16:36:29.453527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.453489 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:29.460390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.460362 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" Nov 03 16:36:29.460677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.460385 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp" event={"ID":"9a24b991-0fff-4c98-8cdf-bbcfee1c261e","Type":"ContainerDied","Data":"81bcd959bac3aa40ae898e2454ddc68b47e3842acc71398e4ad9e84f0e414c79"} Nov 03 16:36:29.460899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.460882 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81bcd959bac3aa40ae898e2454ddc68b47e3842acc71398e4ad9e84f0e414c79" Nov 03 16:36:29.461726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.461702 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e998bdb4-rqtr9"] Nov 03 16:36:29.661247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:29.651789 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:30.009676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:30.009643 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020f3135-815f-45b8-a4e5-66d9a54e4366" path="/var/lib/kubelet/pods/020f3135-815f-45b8-a4e5-66d9a54e4366/volumes" Nov 03 16:36:30.571251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:30.571201 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:30.761991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:30.761962 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:32.652221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:32.652184 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:34.668403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:34.668367 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:35.578807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:35.578770 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:35.777275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:35.777240 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:38.866398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:38.866349 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:39.085934 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:39.085898 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w"] Nov 03 16:36:39.138618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:39.138520 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d968fd-rz49w"] Nov 03 16:36:40.011334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:40.011289 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76923b62-022f-4ab6-a3bd-66b431630e66" path="/var/lib/kubelet/pods/76923b62-022f-4ab6-a3bd-66b431630e66/volumes" Nov 03 16:36:40.403024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:40.402933 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:40.660124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:40.660020 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:40.846652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:40.846618 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4"] Nov 03 16:36:40.861730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:40.861703 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:40.897296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:40.897266 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b774106-kr6c4"] Nov 03 16:36:42.006508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:42.006480 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924c3616-7905-483f-b43d-3859858a80b9" path="/var/lib/kubelet/pods/924c3616-7905-483f-b43d-3859858a80b9/volumes" Nov 03 16:36:42.600245 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:36:42.600172 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:36:42.638372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:42.638336 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:43.316655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:43.316616 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:44.072369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:44.072274 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:44.321285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:44.321253 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:45.913540 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:45.913461 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:46.123194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:46.123162 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:47.525703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:47.525667 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:47.757891 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:47.757854 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:50.798869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:50.798835 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:51.003867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:51.003819 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:51.357150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:51.357119 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:52.065112 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:52.065079 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs"] Nov 03 16:36:52.115996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:52.115962 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f690c3c-n6dqs"] Nov 03 16:36:53.590850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.590815 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k"] Nov 03 16:36:53.591899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.591872 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9a24b991-0fff-4c98-8cdf-bbcfee1c261e" containerName="revoke-keys" Nov 03 16:36:53.591899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.591898 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a24b991-0fff-4c98-8cdf-bbcfee1c261e" containerName="revoke-keys" Nov 03 16:36:53.592163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.592072 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a24b991-0fff-4c98-8cdf-bbcfee1c261e" containerName="revoke-keys" Nov 03 16:36:53.596757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.596732 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.657763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.657730 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k"] Nov 03 16:36:53.678150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.678110 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.678584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.678556 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zdf\" (UniqueName: \"kubernetes.io/projected/a9c55f16-1164-40c8-a946-8ed2960e4624-kube-api-access-n2zdf\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.678834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.678815 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.704275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.702915 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:36:53.704275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.702944 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:36:53.704275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.703120 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:36:53.704275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.703640 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:36:53.780150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.780108 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.780410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.780229 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zdf\" (UniqueName: \"kubernetes.io/projected/a9c55f16-1164-40c8-a946-8ed2960e4624-kube-api-access-n2zdf\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.780410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.780343 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.781030 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.781006 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.783450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.783424 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.843107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.843034 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zdf\" (UniqueName: \"kubernetes.io/projected/a9c55f16-1164-40c8-a946-8ed2960e4624-kube-api-access-n2zdf\") pod \"maas-api-revoke-sub-5f29cd44-d6v4k\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:53.916953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:53.916915 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:54.014601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:54.010689 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc82628-fec8-48be-8d95-8419851d5e31" path="/var/lib/kubelet/pods/3bc82628-fec8-48be-8d95-8419851d5e31/volumes" Nov 03 16:36:54.064293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:54.064257 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292"] Nov 03 16:36:54.069423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:54.069392 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b01fde-qq292"] Nov 03 16:36:54.318287 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:36:54.318254 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c55f16_1164_40c8_a946_8ed2960e4624.slice/crio-2f474aa3166dc53817641f75e23041ba82e006bcf27a52336cd45339e7d9766a WatchSource:0}: Error finding container 2f474aa3166dc53817641f75e23041ba82e006bcf27a52336cd45339e7d9766a: Status 404 returned error can't find the container with id 2f474aa3166dc53817641f75e23041ba82e006bcf27a52336cd45339e7d9766a Nov 03 16:36:54.320201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:54.320176 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k"] Nov 03 16:36:54.706052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:54.706016 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" event={"ID":"a9c55f16-1164-40c8-a946-8ed2960e4624","Type":"ContainerStarted","Data":"1ebf51d1985096423d3aeea51162b20f32c62384be6fabeea6f2547c0b5eb0b5"} Nov 03 16:36:54.706482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:54.706078 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" event={"ID":"a9c55f16-1164-40c8-a946-8ed2960e4624","Type":"ContainerStarted","Data":"2f474aa3166dc53817641f75e23041ba82e006bcf27a52336cd45339e7d9766a"} Nov 03 16:36:54.828616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:54.828562 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" podStartSLOduration=1.828547141 podStartE2EDuration="1.828547141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:54.826052491 +0000 UTC m=+1484.947105340" watchObservedRunningTime="2026-11-03 16:36:54.828547141 +0000 UTC m=+1484.949599964" Nov 03 16:36:55.715699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:55.715660 5946 generic.go:356] "Generic (PLEG): container finished" podID="a9c55f16-1164-40c8-a946-8ed2960e4624" containerID="1ebf51d1985096423d3aeea51162b20f32c62384be6fabeea6f2547c0b5eb0b5" exitCode=0 Nov 03 16:36:55.716127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:55.715740 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" event={"ID":"a9c55f16-1164-40c8-a946-8ed2960e4624","Type":"ContainerDied","Data":"1ebf51d1985096423d3aeea51162b20f32c62384be6fabeea6f2547c0b5eb0b5"} Nov 03 16:36:56.006130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:56.006086 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe1369b-d064-40fc-9dd1-47519c1a9807" path="/var/lib/kubelet/pods/abe1369b-d064-40fc-9dd1-47519c1a9807/volumes" Nov 03 16:36:56.064904 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:56.064871 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw"] Nov 03 16:36:56.069411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:56.069387 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fec2afb-phvgw"] Nov 03 16:36:56.099848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:56.099802 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:56.895434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:56.895408 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:57.029585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.029549 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-openshift-ca\") pod \"a9c55f16-1164-40c8-a946-8ed2960e4624\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " Nov 03 16:36:57.029776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.029619 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c55f16-1164-40c8-a946-8ed2960e4624-kube-api-access-n2zdf\" (UniqueName: \"kubernetes.io/projected/a9c55f16-1164-40c8-a946-8ed2960e4624-kube-api-access-n2zdf\") pod \"a9c55f16-1164-40c8-a946-8ed2960e4624\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " Nov 03 16:36:57.029862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.029786 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-xks-ca\") pod \"a9c55f16-1164-40c8-a946-8ed2960e4624\" (UID: \"a9c55f16-1164-40c8-a946-8ed2960e4624\") " Nov 03 16:36:57.030027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.029996 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-openshift-ca" pod "a9c55f16-1164-40c8-a946-8ed2960e4624" (UID: "a9c55f16-1164-40c8-a946-8ed2960e4624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:36:57.030675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.030648 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:57.032519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.032494 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-xks-ca" pod "a9c55f16-1164-40c8-a946-8ed2960e4624" (UID: "a9c55f16-1164-40c8-a946-8ed2960e4624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:36:57.032625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.032498 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c55f16-1164-40c8-a946-8ed2960e4624-kube-api-access-n2zdf" pod "a9c55f16-1164-40c8-a946-8ed2960e4624" (UID: "a9c55f16-1164-40c8-a946-8ed2960e4624"). InnerVolumeSpecName "kube-api-access-n2zdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:36:57.132388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.132341 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zdf\" (UniqueName: \"kubernetes.io/projected/a9c55f16-1164-40c8-a946-8ed2960e4624-kube-api-access-n2zdf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:57.132388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.132373 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c55f16-1164-40c8-a946-8ed2960e4624-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:36:57.735102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.735064 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" event={"ID":"a9c55f16-1164-40c8-a946-8ed2960e4624","Type":"ContainerDied","Data":"2f474aa3166dc53817641f75e23041ba82e006bcf27a52336cd45339e7d9766a"} Nov 03 16:36:57.735102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.735085 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k" Nov 03 16:36:57.735340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:57.735120 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f474aa3166dc53817641f75e23041ba82e006bcf27a52336cd45339e7d9766a" Nov 03 16:36:58.008684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:36:58.008334 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bed10d1-6b75-44fe-adaa-0c257a008615" path="/var/lib/kubelet/pods/9bed10d1-6b75-44fe-adaa-0c257a008615/volumes" Nov 03 16:37:00.834042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:00.831662 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:01.212539 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:01.212449 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:04.372683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:04.372650 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:05.105125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:05.105095 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:06.664919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:06.664888 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:06.913264 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:06.913173 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:07.146386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:07.146346 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:07.148407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:07.148385 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt"] Nov 03 16:37:07.197805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:07.197772 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6def865-zspjt"] Nov 03 16:37:07.246992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:07.246950 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft"] Nov 03 16:37:07.247411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:07.247348 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" containerID="cri-o://fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6" gracePeriod=30 Nov 03 16:37:07.418289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:07.417097 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:08.006028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.005989 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8856be18-f4ac-4430-91f1-b0ba2283c09b" path="/var/lib/kubelet/pods/8856be18-f4ac-4430-91f1-b0ba2283c09b/volumes" Nov 03 16:37:08.247148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.247119 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:08.464348 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.464269 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:08.906664 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.906628 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2"] Nov 03 16:37:08.907318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.907303 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a9c55f16-1164-40c8-a946-8ed2960e4624" containerName="revoke-keys" Nov 03 16:37:08.907400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.907319 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c55f16-1164-40c8-a946-8ed2960e4624" containerName="revoke-keys" Nov 03 16:37:08.907489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.907479 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c55f16-1164-40c8-a946-8ed2960e4624" containerName="revoke-keys" Nov 03 16:37:08.912542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:08.912524 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.040871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.040822 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.040876 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.040908 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.041065 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.041271 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.041343 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5wj2\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-kube-api-access-b5wj2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.041371 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.041431 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.041600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.041497 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.141406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.141370 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2"] Nov 03 16:37:09.143679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143632 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.143831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143729 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wj2\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-kube-api-access-b5wj2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.143831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143764 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.143831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143817 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143882 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143937 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143963 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.143990 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.144108 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.144398 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.144425 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.144845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.144824 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.145053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.145031 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.145532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.145507 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.148266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.148195 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.164613 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.164511 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.206980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.206945 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.207141 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.207005 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wj2\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-kube-api-access-b5wj2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wrvv2\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.227307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.227275 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:09.577601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.577551 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:09.676955 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:37:09.676926 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb122cee5_df0f_4892_b7b4_af92b6f3afa8.slice/crio-c2df6729fc9d42017c5439ab35a5c6a7090a69c8dfd39fd6d0da7fa6ec78a0ec WatchSource:0}: Error finding container c2df6729fc9d42017c5439ab35a5c6a7090a69c8dfd39fd6d0da7fa6ec78a0ec: Status 404 returned error can't find the container with id c2df6729fc9d42017c5439ab35a5c6a7090a69c8dfd39fd6d0da7fa6ec78a0ec Nov 03 16:37:09.678361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.678339 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2"] Nov 03 16:37:09.680159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.680118 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:09.680324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.680227 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:09.680324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.680274 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:09.778953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.778914 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:09.849802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.849694 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" event={"ID":"b122cee5-df0f-4892-b7b4-af92b6f3afa8","Type":"ContainerStarted","Data":"86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f"} Nov 03 16:37:09.849802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.849788 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" event={"ID":"b122cee5-df0f-4892-b7b4-af92b6f3afa8","Type":"ContainerStarted","Data":"c2df6729fc9d42017c5439ab35a5c6a7090a69c8dfd39fd6d0da7fa6ec78a0ec"} Nov 03 16:37:09.980896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:09.980842 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podStartSLOduration=1.980829682 podStartE2EDuration="1.980829682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:09.977616038 +0000 UTC m=+1500.098668860" watchObservedRunningTime="2026-11-03 16:37:09.980829682 +0000 UTC m=+1500.101882502" Nov 03 16:37:10.047572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.047535 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:10.204146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.204034 5946 patch_prober.go:29] interesting pod/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:37:10.204146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.204131 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:37:10.227520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.227492 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:10.228999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.228969 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:10.229138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.229021 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:10.235274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.235241 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:37:10.240069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.240051 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:37:10.648578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.648548 5946 scope.go:122] "RemoveContainer" containerID="dc1a36c1a6b4f0d56a112c5ae0c1d52c7695c624727612c9062d68ff88a272bc" Nov 03 16:37:10.662574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:10.662548 5946 scope.go:122] "RemoveContainer" containerID="67af41cdea1f68edd113b2fbae1500d2331c855514bd951e7c9f78706e513ca3" Nov 03 16:37:11.239251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:11.238318 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:11.239251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:11.238381 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:12.145589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.145550 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn"] Nov 03 16:37:12.151521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.151424 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.170094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.170067 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn"] Nov 03 16:37:12.211100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.211056 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:37:12.211100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.211086 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:37:12.247233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.247100 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gft\" (UniqueName: \"kubernetes.io/projected/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-kube-api-access-p7gft\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.247233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.247138 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.247808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.247187 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:12.247808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.247331 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:12.247808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.247395 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.277859 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.277824 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:37:12.278021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.277806 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:37:12.285393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.285370 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:37:12.305858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.305830 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:12.350981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.349853 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gft\" (UniqueName: \"kubernetes.io/projected/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-kube-api-access-p7gft\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.350981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.349901 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.350981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.349949 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.351330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.351103 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.360290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.360144 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.440928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.440780 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gft\" (UniqueName: \"kubernetes.io/projected/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-kube-api-access-p7gft\") pod \"maas-api-revoke-sub-59f5a6d8-sxrhn\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.466358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.466325 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:12.648471 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:37:12.640432 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:37:12.668287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.668259 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:37:12.756372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756339 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-socket\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.756582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756401 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-certs\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.756582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756458 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-credential-socket\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.756582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756521 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-token\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-token\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.756786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756580 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-kube-api-access-d2vdf\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-kube-api-access-d2vdf\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.756786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756710 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-socket" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:12.756786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756741 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-certs" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:12.756786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756775 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-podinfo\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.757004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756821 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-envoy\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.757004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756851 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-credential-socket" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:12.757004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.756868 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d12746a-0441-43a7-b3a0-18ce49c446ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d12746a-0441-43a7-b3a0-18ce49c446ea-istiod-ca-cert\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.757158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757011 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-data\") pod \"3d12746a-0441-43a7-b3a0-18ce49c446ea\" (UID: \"3d12746a-0441-43a7-b3a0-18ce49c446ea\") " Nov 03 16:37:12.757310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757286 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12746a-0441-43a7-b3a0-18ce49c446ea-istiod-ca-cert" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:12.757575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757516 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-data" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:12.757782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757765 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d12746a-0441-43a7-b3a0-18ce49c446ea-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.757851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757785 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.757851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757794 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.757851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757803 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.757851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.757814 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.759225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.759177 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-podinfo" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:37:12.759401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.759383 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-token" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:12.759539 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.759521 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-envoy" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:12.759599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.759532 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-kube-api-access-d2vdf" pod "3d12746a-0441-43a7-b3a0-18ce49c446ea" (UID: "3d12746a-0441-43a7-b3a0-18ce49c446ea"). InnerVolumeSpecName "kube-api-access-d2vdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:12.874233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.871970 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.874233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.872017 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.874233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.872033 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:12.874233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:12.872047 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vdf\" (UniqueName: \"kubernetes.io/projected/3d12746a-0441-43a7-b3a0-18ce49c446ea-kube-api-access-d2vdf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:13.027412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.027320 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn"] Nov 03 16:37:13.083227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.079818 5946 generic.go:356] "Generic (PLEG): container finished" podID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerID="fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6" exitCode=0 Nov 03 16:37:13.083227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.079932 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" Nov 03 16:37:13.083227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.080784 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9"] Nov 03 16:37:13.083227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.080814 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" event={"ID":"3d12746a-0441-43a7-b3a0-18ce49c446ea","Type":"ContainerDied","Data":"fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6"} Nov 03 16:37:13.083227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.080846 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft" event={"ID":"3d12746a-0441-43a7-b3a0-18ce49c446ea","Type":"ContainerDied","Data":"c57afe1c94249bdc0afc09a86610bc790c01fd73ec376d59db41947e942e8f8e"} Nov 03 16:37:13.083227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.080865 5946 scope.go:122] "RemoveContainer" containerID="fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6" Nov 03 16:37:13.087069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.087029 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca2d42-ghws9"] Nov 03 16:37:13.105929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.105902 5946 scope.go:122] "RemoveContainer" containerID="09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d" Nov 03 16:37:13.137445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.133404 5946 scope.go:122] "RemoveContainer" containerID="fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6" Nov 03 16:37:13.137445 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:37:13.133849 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6\": container with ID starting with fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6 not found: ID does not exist" containerID="fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6" Nov 03 16:37:13.137445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.133887 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6"} err="failed to get container status \"fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6\": rpc error: code = NotFound desc = could not find container \"fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6\": container with ID starting with fd8eadb0ecc161e92f1db646b718cd14212368ba2cbee3710ea64d861e8724b6 not found: ID does not exist" Nov 03 16:37:13.137445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.133911 5946 scope.go:122] "RemoveContainer" containerID="09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d" Nov 03 16:37:13.137445 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:37:13.134233 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d\": container with ID starting with 09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d not found: ID does not exist" containerID="09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d" Nov 03 16:37:13.137445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.134261 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d"} err="failed to get container status \"09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d\": rpc error: code = NotFound desc = could not find container \"09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d\": container with ID starting with 09e5f533fb06443c8d0eb88a710bcf92db3ef299ae9d163f82be6a47b2c8014d not found: ID does not exist" Nov 03 16:37:13.238236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.236123 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:13.238236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.236183 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:13.257790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.257753 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft"] Nov 03 16:37:13.269826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.269794 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e6def865-gw-openshift-default-57768c7645-6j2ft"] Nov 03 16:37:13.644043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:13.644010 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:14.073967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:14.073926 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" path="/var/lib/kubelet/pods/3d12746a-0441-43a7-b3a0-18ce49c446ea/volumes" Nov 03 16:37:14.074658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:14.074632 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c69fdb-6180-4cda-8d9c-05ce011d44cd" path="/var/lib/kubelet/pods/b6c69fdb-6180-4cda-8d9c-05ce011d44cd/volumes" Nov 03 16:37:14.091687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:14.091657 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" event={"ID":"90ec9cf2-6a82-42ee-9a65-8b467d52fad1","Type":"ContainerStarted","Data":"5ab48f6601eaf6273097249ea28c010998a0e339f2747067e3f8266eba25f924"} Nov 03 16:37:14.091855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:14.091702 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" event={"ID":"90ec9cf2-6a82-42ee-9a65-8b467d52fad1","Type":"ContainerStarted","Data":"d0c6d36e2afca67acda9bc9c45be5609e3e55b99e594b431f38c1841ac426efc"} Nov 03 16:37:14.244235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:14.243307 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:14.244235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:14.243369 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:14.254247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:14.254113 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" podStartSLOduration=2.25409303 podStartE2EDuration="2.25409303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:14.244049202 +0000 UTC m=+1504.365102027" watchObservedRunningTime="2026-11-03 16:37:14.25409303 +0000 UTC m=+1504.375145854" Nov 03 16:37:15.101265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:15.101234 5946 generic.go:356] "Generic (PLEG): container finished" podID="90ec9cf2-6a82-42ee-9a65-8b467d52fad1" containerID="5ab48f6601eaf6273097249ea28c010998a0e339f2747067e3f8266eba25f924" exitCode=0 Nov 03 16:37:15.101812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:15.101785 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" event={"ID":"90ec9cf2-6a82-42ee-9a65-8b467d52fad1","Type":"ContainerDied","Data":"5ab48f6601eaf6273097249ea28c010998a0e339f2747067e3f8266eba25f924"} Nov 03 16:37:15.249234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:15.247376 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:15.249234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:15.247435 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:16.063579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.063542 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x"] Nov 03 16:37:16.118659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.117230 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa96adc-k9h6x"] Nov 03 16:37:16.264102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.263263 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:16.264102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.263322 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:16.356850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.356817 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:16.465134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.465095 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-openshift-ca\") pod \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " Nov 03 16:37:16.465389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.465282 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-xks-ca\") pod \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " Nov 03 16:37:16.465389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.465334 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-kube-api-access-p7gft\" (UniqueName: \"kubernetes.io/projected/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-kube-api-access-p7gft\") pod \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\" (UID: \"90ec9cf2-6a82-42ee-9a65-8b467d52fad1\") " Nov 03 16:37:16.465537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.465505 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-openshift-ca" pod "90ec9cf2-6a82-42ee-9a65-8b467d52fad1" (UID: "90ec9cf2-6a82-42ee-9a65-8b467d52fad1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:16.466052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.466029 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:16.468085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.468056 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-xks-ca" pod "90ec9cf2-6a82-42ee-9a65-8b467d52fad1" (UID: "90ec9cf2-6a82-42ee-9a65-8b467d52fad1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:37:16.468241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.468124 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-kube-api-access-p7gft" pod "90ec9cf2-6a82-42ee-9a65-8b467d52fad1" (UID: "90ec9cf2-6a82-42ee-9a65-8b467d52fad1"). InnerVolumeSpecName "kube-api-access-p7gft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:16.567943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.567900 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:16.567943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:16.567941 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gft\" (UniqueName: \"kubernetes.io/projected/90ec9cf2-6a82-42ee-9a65-8b467d52fad1-kube-api-access-p7gft\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:17.151153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:17.151113 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" event={"ID":"90ec9cf2-6a82-42ee-9a65-8b467d52fad1","Type":"ContainerDied","Data":"d0c6d36e2afca67acda9bc9c45be5609e3e55b99e594b431f38c1841ac426efc"} Nov 03 16:37:17.151691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:17.151168 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c6d36e2afca67acda9bc9c45be5609e3e55b99e594b431f38c1841ac426efc" Nov 03 16:37:17.151691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:17.151129 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn" Nov 03 16:37:17.232235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:17.228724 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 16:37:17.232235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:17.228804 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 16:37:17.812541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:17.812500 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:18.023643 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:18.023601 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420f1a53-6904-4770-9571-b1dbe9f38248" path="/var/lib/kubelet/pods/420f1a53-6904-4770-9571-b1dbe9f38248/volumes" Nov 03 16:37:18.888020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:18.887984 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:18.888752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:18.888731 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:18.889711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:18.889693 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:37:19.063806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:19.063773 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt"] Nov 03 16:37:19.145771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:19.145682 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln"] Nov 03 16:37:19.150940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:19.150912 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab4ec34-b4ppt"] Nov 03 16:37:19.155219 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:19.155182 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634928da-68wln"] Nov 03 16:37:20.006378 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:20.006338 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994743d0-eb54-44cc-a49b-45cf570a96c5" path="/var/lib/kubelet/pods/994743d0-eb54-44cc-a49b-45cf570a96c5/volumes" Nov 03 16:37:20.007100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:20.007079 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3ef70d-72b3-4f10-8791-1232d2b2fa94" path="/var/lib/kubelet/pods/da3ef70d-72b3-4f10-8791-1232d2b2fa94/volumes" Nov 03 16:37:22.665081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:22.665026 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:22.857958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:22.857928 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:23.897294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.897264 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn"] Nov 03 16:37:23.897967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.897952 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" Nov 03 16:37:23.898012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.897968 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" Nov 03 16:37:23.898049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.898012 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" Nov 03 16:37:23.898049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.898017 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" Nov 03 16:37:23.898049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.898038 5946 cpu_manager.go:414] "containerMap: removing container" podUID="90ec9cf2-6a82-42ee-9a65-8b467d52fad1" containerName="revoke-keys" Nov 03 16:37:23.898049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.898044 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ec9cf2-6a82-42ee-9a65-8b467d52fad1" containerName="revoke-keys" Nov 03 16:37:23.898180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.898169 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" Nov 03 16:37:23.898252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.898179 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ec9cf2-6a82-42ee-9a65-8b467d52fad1" containerName="revoke-keys" Nov 03 16:37:23.900780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.900764 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.929675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.929647 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.929795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.929704 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.929837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.929786 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.929927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.929914 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnms\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-kube-api-access-ggnms\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.929969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.929950 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.930002 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.929992 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.930061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.930049 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.930136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.930122 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:23.930170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:23.930156 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.011571 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.011537 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn"] Nov 03 16:37:24.032118 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032078 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnms\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-kube-api-access-ggnms\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032149 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032264 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032496 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032604 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032654 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032720 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032788 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032850 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.032915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.032892 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.033420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.033400 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.033658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.033632 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.033977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.033952 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.034086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.034024 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.036403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.036382 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.036485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.036400 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.093438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.093407 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.093592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.093485 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnms\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-kube-api-access-ggnms\") pod \"maas-default-gateway-openshift-default-5bc884dd87-n7tnn\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.216073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.215990 5946 generic.go:356] "Generic (PLEG): container finished" podID="523fd694-634d-4bd0-b573-773d9e42feeb" containerID="7c1123317bc485eb77f316f2bd3c225dc026166e13a7605f2b48f96b66f0d053" exitCode=0 Nov 03 16:37:24.216073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.216047 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" event={"ID":"523fd694-634d-4bd0-b573-773d9e42feeb","Type":"ContainerDied","Data":"7c1123317bc485eb77f316f2bd3c225dc026166e13a7605f2b48f96b66f0d053"} Nov 03 16:37:24.216341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.216187 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:24.216527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.216510 5946 scope.go:122] "RemoveContainer" containerID="7c1123317bc485eb77f316f2bd3c225dc026166e13a7605f2b48f96b66f0d053" Nov 03 16:37:24.218356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.218325 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:24.218481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.218396 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:24.218481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.218432 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:24.661522 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:37:24.661484 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7e77e0_0571_4236_b6c4_a5a7744ff9b8.slice/crio-a587c64783bcd90ac00a8de32894a573e553073672e33483fe20b4c39be109e4 WatchSource:0}: Error finding container a587c64783bcd90ac00a8de32894a573e553073672e33483fe20b4c39be109e4: Status 404 returned error can't find the container with id a587c64783bcd90ac00a8de32894a573e553073672e33483fe20b4c39be109e4 Nov 03 16:37:24.664726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.664698 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn"] Nov 03 16:37:24.664879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.664844 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:24.664954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.664925 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:24.664999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:24.664984 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:25.246879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:25.246841 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" event={"ID":"523fd694-634d-4bd0-b573-773d9e42feeb","Type":"ContainerStarted","Data":"0323f16947daee1b6ec0f5ccd42939be892ed73b41c127bb7e040cd6852db8be"} Nov 03 16:37:25.250330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:25.250294 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" event={"ID":"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8","Type":"ContainerStarted","Data":"a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3"} Nov 03 16:37:25.250482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:25.250353 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" event={"ID":"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8","Type":"ContainerStarted","Data":"a587c64783bcd90ac00a8de32894a573e553073672e33483fe20b4c39be109e4"} Nov 03 16:37:25.505109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:25.504995 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podStartSLOduration=2.504979049 podStartE2EDuration="2.504979049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:25.504595329 +0000 UTC m=+1515.625648152" watchObservedRunningTime="2026-11-03 16:37:25.504979049 +0000 UTC m=+1515.626031875" Nov 03 16:37:25.548709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:25.548674 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:37:25.573291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:25.573253 5946 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:37:25.573658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:25.573611 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 16:37:26.222234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:26.219422 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:26.222234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:26.219558 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 16:37:26.222234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:26.219600 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 16:37:26.543234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:26.542676 5946 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:37:26.543234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:26.542742 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 16:37:26.984065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:26.983929 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:27.231233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:27.231179 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 16:37:27.231500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:27.231468 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 16:37:27.592234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:27.582184 5946 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:37:27.592234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:27.582272 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 16:37:28.225819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:28.225774 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 16:37:28.226171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:28.226122 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 16:37:28.558047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:28.558012 5946 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:37:28.558374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:28.558343 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 16:37:28.559478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:28.559453 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:28.837288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:28.835001 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:28.923459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:28.923425 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:37:29.028236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.027845 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp"] Nov 03 16:37:29.044447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.044400 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:29.080650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.080606 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-161f6056-r4bcp"] Nov 03 16:37:29.190935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.174334 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9"] Nov 03 16:37:29.190935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.174584 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" containerID="cri-o://728dd2b5a6be80e249bb0eda2e62709bcb1f65a3ff6236caaeb22340bae7b41a" gracePeriod=30 Nov 03 16:37:29.243231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.241996 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 16:37:29.243231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.242052 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 16:37:29.270244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.265398 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:29.547232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.545831 5946 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:37:29.547232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:29.545890 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 16:37:30.007116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.007084 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a24b991-0fff-4c98-8cdf-bbcfee1c261e" path="/var/lib/kubelet/pods/9a24b991-0fff-4c98-8cdf-bbcfee1c261e/volumes" Nov 03 16:37:30.196537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.196499 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:30.217701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.217664 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 16:37:30.217991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.217958 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 16:37:30.407711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.407634 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:30.544793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.544754 5946 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:37:30.545123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.545076 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 16:37:30.764475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:30.764438 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:31.074941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:31.074866 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb"] Nov 03 16:37:31.141454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:31.141416 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1daea440-q56nb"] Nov 03 16:37:31.232884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:31.232508 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 16:37:31.232884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:31.232566 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 16:37:31.547198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:31.547171 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:37:31.548501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:31.548481 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 16:37:31.760661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:31.760624 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:32.007594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:32.007560 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b091a762-284d-4410-9a11-089dd21e0b70" path="/var/lib/kubelet/pods/b091a762-284d-4410-9a11-089dd21e0b70/volumes" Nov 03 16:37:32.217333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:32.217298 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 16:37:32.217742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:32.217353 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 16:37:32.981096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:32.981058 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:33.730883 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:33.730852 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:33.731416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:33.731236 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:33.732087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:33.732066 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:37:33.835874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:33.835842 5946 patch_prober.go:29] interesting pod/e2e-derive-161f6056-openshift-default-c456fd685-trmm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:37:33.836040 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:33.835893 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:37:34.369637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.369527 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:34.525713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.525680 5946 generic.go:356] "Generic (PLEG): container finished" podID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerID="728dd2b5a6be80e249bb0eda2e62709bcb1f65a3ff6236caaeb22340bae7b41a" exitCode=0 Nov 03 16:37:34.526003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.525974 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" event={"ID":"f8e5925d-2e74-4a38-a86b-5bbb45424268","Type":"ContainerDied","Data":"728dd2b5a6be80e249bb0eda2e62709bcb1f65a3ff6236caaeb22340bae7b41a"} Nov 03 16:37:34.526081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.526044 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" event={"ID":"f8e5925d-2e74-4a38-a86b-5bbb45424268","Type":"ContainerDied","Data":"192dc804bcf6f6be3226c2733c4eb04c31a32aee20019b9db63cb0b5d9c6daa4"} Nov 03 16:37:34.526081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.526068 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="192dc804bcf6f6be3226c2733c4eb04c31a32aee20019b9db63cb0b5d9c6daa4" Nov 03 16:37:34.526191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.526087 5946 scope.go:122] "RemoveContainer" containerID="6a019ba89d568f8299b1030aa2adc8349f53f36b4f8f0b87b8d5c69b677e2c98" Nov 03 16:37:34.543096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.543081 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:37:34.639029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.638993 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-envoy\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639067 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-data\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639114 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-socket\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639141 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e5925d-2e74-4a38-a86b-5bbb45424268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e5925d-2e74-4a38-a86b-5bbb45424268-istiod-ca-cert\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639228 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-kube-api-access-8hcfj\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-kube-api-access-8hcfj\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639289 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-podinfo\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639345 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-data" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:34.639478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639364 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-token\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-token\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639461 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-socket" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:34.639689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639506 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-credential-socket\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639559 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-certs\") pod \"f8e5925d-2e74-4a38-a86b-5bbb45424268\" (UID: \"f8e5925d-2e74-4a38-a86b-5bbb45424268\") " Nov 03 16:37:34.639689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639607 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e5925d-2e74-4a38-a86b-5bbb45424268-istiod-ca-cert" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:34.639920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639884 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-credential-socket" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:34.639920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.639908 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-certs" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:34.640806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.640771 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.640806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.640800 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.640969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.640816 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e5925d-2e74-4a38-a86b-5bbb45424268-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.640969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.640830 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.640969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.640843 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.641715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.641687 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-envoy" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:37:34.641808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.641779 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-token" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:34.641870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.641830 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-kube-api-access-8hcfj" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "kube-api-access-8hcfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:34.641870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.641845 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-podinfo" pod "f8e5925d-2e74-4a38-a86b-5bbb45424268" (UID: "f8e5925d-2e74-4a38-a86b-5bbb45424268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:37:34.742677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.742634 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.742677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.742667 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hcfj\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-kube-api-access-8hcfj\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.742677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.742683 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:34.743186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:34.742695 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e5925d-2e74-4a38-a86b-5bbb45424268-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:35.535158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:35.535124 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9" Nov 03 16:37:35.834040 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:35.833962 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9"] Nov 03 16:37:35.842614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:35.842589 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-161f6056-openshift-default-c456fd685-trmm9"] Nov 03 16:37:36.005754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:36.005718 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" path="/var/lib/kubelet/pods/f8e5925d-2e74-4a38-a86b-5bbb45424268/volumes" Nov 03 16:37:38.067229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.067174 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf"] Nov 03 16:37:38.118415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.118375 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2351bf03-ppdcf"] Nov 03 16:37:38.310424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.310392 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl"] Nov 03 16:37:38.311116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.311096 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" Nov 03 16:37:38.311237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.311118 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" Nov 03 16:37:38.311347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.311332 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" Nov 03 16:37:38.311407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.311361 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d12746a-0441-43a7-b3a0-18ce49c446ea" containerName="istio-proxy" Nov 03 16:37:38.314920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.314896 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.352195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.352155 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.352408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.352381 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drl2s\" (UniqueName: \"kubernetes.io/projected/f425dc5d-e618-41f4-a1ea-963a3722c1cc-kube-api-access-drl2s\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.352464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.352428 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.377336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.377308 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:37:38.377570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.377335 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:37:38.377570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.377399 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl"] Nov 03 16:37:38.377791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.377431 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:37:38.378547 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.378526 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:37:38.454837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.454801 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.454986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.454959 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drl2s\" (UniqueName: \"kubernetes.io/projected/f425dc5d-e618-41f4-a1ea-963a3722c1cc-kube-api-access-drl2s\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.455038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.454995 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.455701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.455666 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.457648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.457626 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.515862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.515824 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drl2s\" (UniqueName: \"kubernetes.io/projected/f425dc5d-e618-41f4-a1ea-963a3722c1cc-kube-api-access-drl2s\") pod \"maas-api-revoke-keys-e2e-mt-06f127ec-55stl\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.637288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.636483 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:38.673091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.673050 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:38.892862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:38.892698 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:39.535245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:39.513440 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl"] Nov 03 16:37:39.587245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:39.585165 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:39.836289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:39.835708 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:39.945258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:39.934574 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" event={"ID":"f425dc5d-e618-41f4-a1ea-963a3722c1cc","Type":"ContainerStarted","Data":"c06dc2e5674ecbdd2fc72b94ad2ee8f6e6174f7699c175bf29959f4264f90853"} Nov 03 16:37:40.029676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.029630 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:40.084567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.084536 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc142a9-f3c9-408b-9b51-10578efd1196" path="/var/lib/kubelet/pods/1bc142a9-f3c9-408b-9b51-10578efd1196/volumes" Nov 03 16:37:40.243744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.243638 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:40.687550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.687457 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687"] Nov 03 16:37:40.689376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.689316 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" Nov 03 16:37:40.689558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.689542 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" Nov 03 16:37:40.689890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.689875 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e5925d-2e74-4a38-a86b-5bbb45424268" containerName="istio-proxy" Nov 03 16:37:40.693483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.693456 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.847022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.846988 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9635db0d-openshift-default-dockercfg-v4bpg\"" Nov 03 16:37:40.947376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.899291 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-certs\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.952516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.952482 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istiod-ca-cert\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.957341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.948083 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687"] Nov 03 16:37:40.957731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.910117 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:40.959070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.959025 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-envoy\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.959428 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.959401 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7wx\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-kube-api-access-2f7wx\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.959659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.959644 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-token\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.959805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.959790 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-podinfo\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.960027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.960012 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-data\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.960322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.960305 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-credential-socket\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.960481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.960465 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-socket\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:40.978437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:40.976443 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" event={"ID":"f425dc5d-e618-41f4-a1ea-963a3722c1cc","Type":"ContainerStarted","Data":"031c2f63db4e41209036001a349781a8d1f48e890c7fcf880886976d254fa1de"} Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.122703 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7wx\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-kube-api-access-2f7wx\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.122788 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-token\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.122818 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-podinfo\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.122894 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-data\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.122996 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-credential-socket\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.123030 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-socket\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.123912 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-certs\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.123987 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istiod-ca-cert\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.125861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.124171 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-envoy\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.129996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.127394 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-envoy\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.129996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.127841 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-socket\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.129996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.128079 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-data\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.129996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.128108 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-credential-socket\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.129996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.128375 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-certs\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.129996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.129824 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istiod-ca-cert\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.131249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.131195 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-podinfo\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.189574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.189539 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-token\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.189810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.189635 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7wx\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-kube-api-access-2f7wx\") pod \"e2e-mt-9635db0d-openshift-default-5c876b8797-km687\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.318939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.318898 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:41.403152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.403098 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:41.778574 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:37:41.778541 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bdfccd_eb96_409f_9f80_d4cecc7a6a5f.slice/crio-76fc2d1303ab36afc10f77e2bc6b6283157660d6f8f3dc245553af57ab4079c4 WatchSource:0}: Error finding container 76fc2d1303ab36afc10f77e2bc6b6283157660d6f8f3dc245553af57ab4079c4: Status 404 returned error can't find the container with id 76fc2d1303ab36afc10f77e2bc6b6283157660d6f8f3dc245553af57ab4079c4 Nov 03 16:37:41.781496 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.781438 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" podStartSLOduration=3.7814185719999998 podStartE2EDuration="3.781418572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:41.128806557 +0000 UTC m=+1531.249859380" watchObservedRunningTime="2026-11-03 16:37:41.781418572 +0000 UTC m=+1531.902471396" Nov 03 16:37:41.781848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.781814 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:41.781917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.781902 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:41.781968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.781948 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687"] Nov 03 16:37:41.782023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.781950 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:41.988026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.987994 5946 generic.go:356] "Generic (PLEG): container finished" podID="f425dc5d-e618-41f4-a1ea-963a3722c1cc" containerID="031c2f63db4e41209036001a349781a8d1f48e890c7fcf880886976d254fa1de" exitCode=0 Nov 03 16:37:41.988308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.988285 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" event={"ID":"f425dc5d-e618-41f4-a1ea-963a3722c1cc","Type":"ContainerDied","Data":"031c2f63db4e41209036001a349781a8d1f48e890c7fcf880886976d254fa1de"} Nov 03 16:37:41.990804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.990776 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" event={"ID":"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f","Type":"ContainerStarted","Data":"bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a"} Nov 03 16:37:41.990925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:41.990830 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" event={"ID":"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f","Type":"ContainerStarted","Data":"76fc2d1303ab36afc10f77e2bc6b6283157660d6f8f3dc245553af57ab4079c4"} Nov 03 16:37:42.066135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:42.066037 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd"] Nov 03 16:37:42.070578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:42.070548 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f7c92e-xghbd"] Nov 03 16:37:42.233981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:42.233901 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" podStartSLOduration=2.233882284 podStartE2EDuration="2.233882284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:42.23194811 +0000 UTC m=+1532.353000933" watchObservedRunningTime="2026-11-03 16:37:42.233882284 +0000 UTC m=+1532.354935107" Nov 03 16:37:42.319747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:42.319664 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:42.326015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:42.325986 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:42.589527 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:37:42.589427 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:37:43.000661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.000060 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:43.001044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.000937 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:37:43.116792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.116726 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6"] Nov 03 16:37:43.123985 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.123954 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00940418-jxxr6"] Nov 03 16:37:43.194257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.194194 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:43.277360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.277271 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-xks-ca\") pod \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " Nov 03 16:37:43.277360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.277337 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-openshift-ca\") pod \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " Nov 03 16:37:43.277578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.277379 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f425dc5d-e618-41f4-a1ea-963a3722c1cc-kube-api-access-drl2s\" (UniqueName: \"kubernetes.io/projected/f425dc5d-e618-41f4-a1ea-963a3722c1cc-kube-api-access-drl2s\") pod \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\" (UID: \"f425dc5d-e618-41f4-a1ea-963a3722c1cc\") " Nov 03 16:37:43.277795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.277769 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-openshift-ca" pod "f425dc5d-e618-41f4-a1ea-963a3722c1cc" (UID: "f425dc5d-e618-41f4-a1ea-963a3722c1cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:43.277938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.277924 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:43.279710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.279687 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-xks-ca" pod "f425dc5d-e618-41f4-a1ea-963a3722c1cc" (UID: "f425dc5d-e618-41f4-a1ea-963a3722c1cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:37:43.279803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.279747 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f425dc5d-e618-41f4-a1ea-963a3722c1cc-kube-api-access-drl2s" pod "f425dc5d-e618-41f4-a1ea-963a3722c1cc" (UID: "f425dc5d-e618-41f4-a1ea-963a3722c1cc"). InnerVolumeSpecName "kube-api-access-drl2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:43.379800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.379763 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f425dc5d-e618-41f4-a1ea-963a3722c1cc-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:43.379800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:43.379795 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drl2s\" (UniqueName: \"kubernetes.io/projected/f425dc5d-e618-41f4-a1ea-963a3722c1cc-kube-api-access-drl2s\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:37:44.008011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:44.007968 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad2dae2-3cce-4572-94bc-8f67b2837c98" path="/var/lib/kubelet/pods/2ad2dae2-3cce-4572-94bc-8f67b2837c98/volumes" Nov 03 16:37:44.008687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:44.008664 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04b1cde-9c26-4dbb-a090-f3e6120b2f26" path="/var/lib/kubelet/pods/f04b1cde-9c26-4dbb-a090-f3e6120b2f26/volumes" Nov 03 16:37:44.010771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:44.010750 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" Nov 03 16:37:44.010893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:44.010779 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl" event={"ID":"f425dc5d-e618-41f4-a1ea-963a3722c1cc","Type":"ContainerDied","Data":"c06dc2e5674ecbdd2fc72b94ad2ee8f6e6174f7699c175bf29959f4264f90853"} Nov 03 16:37:44.010893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:44.010828 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c06dc2e5674ecbdd2fc72b94ad2ee8f6e6174f7699c175bf29959f4264f90853" Nov 03 16:37:45.070060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:45.070027 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp"] Nov 03 16:37:45.084079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:45.084053 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba9775-4h7jp"] Nov 03 16:37:45.141627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:45.141588 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:45.344278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:45.344185 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:45.553291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:45.553225 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:46.007850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:46.007816 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4903f4-eaee-4f52-9a86-3622ba5d5427" path="/var/lib/kubelet/pods/7d4903f4-eaee-4f52-9a86-3622ba5d5427/volumes" Nov 03 16:37:48.605124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:48.605091 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:48.802523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:48.802480 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:49.317976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.317927 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz"] Nov 03 16:37:49.319386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.319361 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f425dc5d-e618-41f4-a1ea-963a3722c1cc" containerName="revoke-keys" Nov 03 16:37:49.319550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.319535 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f425dc5d-e618-41f4-a1ea-963a3722c1cc" containerName="revoke-keys" Nov 03 16:37:49.319864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.319833 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f425dc5d-e618-41f4-a1ea-963a3722c1cc" containerName="revoke-keys" Nov 03 16:37:49.358504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.358470 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:49.430865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.430366 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz"] Nov 03 16:37:49.432563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.432517 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:49.432891 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.432858 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqfx\" (UniqueName: \"kubernetes.io/projected/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-kube-api-access-6zqfx\") pod \"payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:49.443363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.442629 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9635db0d\"" Nov 03 16:37:49.443363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.442748 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9635db0d-dockercfg-gtsxf\"" Nov 03 16:37:49.534881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.534843 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:49.535099 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.534920 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqfx\" (UniqueName: \"kubernetes.io/projected/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-kube-api-access-6zqfx\") pod \"payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:49.535882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.535857 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:49.653121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.653033 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqfx\" (UniqueName: \"kubernetes.io/projected/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-kube-api-access-6zqfx\") pod \"payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:49.781262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:49.780492 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:50.338338 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:37:50.338310 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b44e7f3_e5a3_4ae1_b285_96f756a2b268.slice/crio-f0a3507423b4f4bc0ca1dc1b6ef8c6e10b716f730d5453b5fa63dc1963ea6d85 WatchSource:0}: Error finding container f0a3507423b4f4bc0ca1dc1b6ef8c6e10b716f730d5453b5fa63dc1963ea6d85: Status 404 returned error can't find the container with id f0a3507423b4f4bc0ca1dc1b6ef8c6e10b716f730d5453b5fa63dc1963ea6d85 Nov 03 16:37:50.338833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:50.338811 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz"] Nov 03 16:37:50.468597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:50.468550 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:51.082318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:51.082174 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" event={"ID":"9b44e7f3-e5a3-4ae1-b285-96f756a2b268","Type":"ContainerStarted","Data":"b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19"} Nov 03 16:37:51.082318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:51.082291 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" event={"ID":"9b44e7f3-e5a3-4ae1-b285-96f756a2b268","Type":"ContainerStarted","Data":"f0a3507423b4f4bc0ca1dc1b6ef8c6e10b716f730d5453b5fa63dc1963ea6d85"} Nov 03 16:37:51.083875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:51.082549 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:37:51.205836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:51.205784 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" podStartSLOduration=2.205769045 podStartE2EDuration="2.205769045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:51.202629663 +0000 UTC m=+1541.323682483" watchObservedRunningTime="2026-11-03 16:37:51.205769045 +0000 UTC m=+1541.326821867" Nov 03 16:37:52.084403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:52.084365 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc"] Nov 03 16:37:52.133856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:52.133821 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99640711-vd2zc"] Nov 03 16:37:52.663266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:52.663188 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:52.873181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:52.873144 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:53.074271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:53.074199 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz"] Nov 03 16:37:53.125000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:53.124967 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7ae4bc-njhrz"] Nov 03 16:37:53.948233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:53.947710 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:54.033412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:54.033376 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfbdc5c-243d-4a5e-a343-1d7e39c842fc" path="/var/lib/kubelet/pods/1dfbdc5c-243d-4a5e-a343-1d7e39c842fc/volumes" Nov 03 16:37:54.034106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:54.034084 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bae6f3-5757-48ac-92f5-441867cb0c28" path="/var/lib/kubelet/pods/40bae6f3-5757-48ac-92f5-441867cb0c28/volumes" Nov 03 16:37:54.141077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:54.141049 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:54.417111 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:54.417063 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:55.569817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.569782 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq"] Nov 03 16:37:55.574810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.574785 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.647047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.647014 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-509f14-openshift-default-dockercfg-zt8jn\"" Nov 03 16:37:55.672286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672248 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-credential-socket\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.672472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672298 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dt5\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-kube-api-access-v2dt5\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.672472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672384 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-token\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.672577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672495 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-certs\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.672689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672673 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-podinfo\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.672753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672727 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-data\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.672799 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672781 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-socket\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.672895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.672853 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-envoy\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.673045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.673020 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267d739d-4e26-432a-bd7f-865b6ccbc8df-istiod-ca-cert\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.686802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.686771 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq"] Nov 03 16:37:55.706305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.706267 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:55.794058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.794015 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-podinfo\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.800721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.800696 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-data\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.801383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.801356 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-socket\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.801867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.801851 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-envoy\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.804997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.801247 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-data\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.804997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.801715 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-socket\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.806014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.805950 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267d739d-4e26-432a-bd7f-865b6ccbc8df-istiod-ca-cert\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.806164 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.806144 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267d739d-4e26-432a-bd7f-865b6ccbc8df-istiod-ca-cert\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.806357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.806328 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-credential-socket\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.808034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.808008 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dt5\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-kube-api-access-v2dt5\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.816625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.816594 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-token\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.817135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.806713 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-credential-socket\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.820341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.809163 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-podinfo\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.820341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.817337 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-certs\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.820341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.818502 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-certs\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.832181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.832147 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-envoy\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.876104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.876068 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dt5\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-kube-api-access-v2dt5\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.877878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.877850 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-token\") pod \"e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.890259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.890234 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:55.902661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:55.902626 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:56.110143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:56.109787 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:56.350785 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:56.347831 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq"] Nov 03 16:37:56.360517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:56.360439 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:56.360517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:56.360511 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:56.360695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:56.360540 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:37:57.074977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:57.074948 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k"] Nov 03 16:37:57.082095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:57.082068 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a459b49-2cd9k"] Nov 03 16:37:57.153954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:57.153920 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" event={"ID":"267d739d-4e26-432a-bd7f-865b6ccbc8df","Type":"ContainerStarted","Data":"e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c"} Nov 03 16:37:57.154130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:57.153982 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" event={"ID":"267d739d-4e26-432a-bd7f-865b6ccbc8df","Type":"ContainerStarted","Data":"d322b9f0bb0dc5fc208986a22fc54aa18722400e9252060d3db3836244ea1602"} Nov 03 16:37:57.305413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:57.305364 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podStartSLOduration=2.305348868 podStartE2EDuration="2.305348868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:57.303940171 +0000 UTC m=+1547.424993005" watchObservedRunningTime="2026-11-03 16:37:57.305348868 +0000 UTC m=+1547.426401691" Nov 03 16:37:57.891048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:57.891009 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:57.896879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:57.896854 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:58.009254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:58.009193 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94df55e4-bbc0-4c88-8ece-92ce1d6f0d58" path="/var/lib/kubelet/pods/94df55e4-bbc0-4c88-8ece-92ce1d6f0d58/volumes" Nov 03 16:37:58.162381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:58.162299 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:37:58.163412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:37:58.163391 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:38:00.097835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:00.097796 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:00.308934 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:00.308868 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:00.550795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:00.550763 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:01.689719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:01.689685 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:02.011258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.011185 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:02.069082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.069048 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm"] Nov 03 16:38:02.095622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.095590 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:38:02.116537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.116504 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78f5709b-wpclm"] Nov 03 16:38:02.201952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.201892 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:02.213453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.213415 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl"] Nov 03 16:38:02.217974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.217948 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06f127ec-55stl"] Nov 03 16:38:02.686439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:02.686410 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:03.065021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:03.064982 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5"] Nov 03 16:38:03.115883 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:03.115852 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60338208-b8zr5"] Nov 03 16:38:03.880311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:03.880275 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59"] Nov 03 16:38:03.880501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:03.880467 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" containerID="cri-o://9f598bc7fe8791d40da0306afc9a9d7a210b5e5b4f47e2da6f89965c45584d1d" gracePeriod=30 Nov 03 16:38:04.014126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:04.014088 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00db8266-7b88-4c8e-920c-0ba95c9b5ccc" path="/var/lib/kubelet/pods/00db8266-7b88-4c8e-920c-0ba95c9b5ccc/volumes" Nov 03 16:38:04.014700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:04.014683 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409b02a4-2bb7-4514-b88f-6ae3356071b7" path="/var/lib/kubelet/pods/409b02a4-2bb7-4514-b88f-6ae3356071b7/volumes" Nov 03 16:38:04.015181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:04.015165 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f425dc5d-e618-41f4-a1ea-963a3722c1cc" path="/var/lib/kubelet/pods/f425dc5d-e618-41f4-a1ea-963a3722c1cc/volumes" Nov 03 16:38:06.332383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:06.332350 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:06.528976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:06.528866 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:07.398804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:07.398726 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:09.262479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.262441 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq"] Nov 03 16:38:09.290665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.290617 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab82594-ldbjq"] Nov 03 16:38:09.325138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.325097 5946 generic.go:356] "Generic (PLEG): container finished" podID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerID="9f598bc7fe8791d40da0306afc9a9d7a210b5e5b4f47e2da6f89965c45584d1d" exitCode=0 Nov 03 16:38:09.325582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.325552 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" event={"ID":"1ec3c61f-ca5f-4bb5-af09-8885570c37e8","Type":"ContainerDied","Data":"9f598bc7fe8791d40da0306afc9a9d7a210b5e5b4f47e2da6f89965c45584d1d"} Nov 03 16:38:09.350278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.350248 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:38:09.427346 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427298 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-socket\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.427644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427613 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-credential-socket\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.427771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427754 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-token\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-token\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.428347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427785 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istiod-ca-cert\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.428347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427819 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-envoy\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.428347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427855 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-podinfo\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.428347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427892 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-data\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.428347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.427924 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-kube-api-access-m6mjn\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-kube-api-access-m6mjn\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.428347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.428038 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-certs\") pod \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\" (UID: \"1ec3c61f-ca5f-4bb5-af09-8885570c37e8\") " Nov 03 16:38:09.428679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.428405 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-data" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:09.428679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.428521 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istiod-ca-cert" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:09.428777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.428687 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-certs" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:09.428777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.428697 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-socket" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:09.429131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.429106 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-credential-socket" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:09.429639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.429613 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.429742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.429644 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.429742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.429660 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.429742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.429673 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.429742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.429687 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.431018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.430991 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-kube-api-access-m6mjn" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "kube-api-access-m6mjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:09.431308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.431284 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-token" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:09.431463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.431432 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-podinfo" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:38:09.431516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.431433 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-envoy" pod "1ec3c61f-ca5f-4bb5-af09-8885570c37e8" (UID: "1ec3c61f-ca5f-4bb5-af09-8885570c37e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:09.531807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.531667 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.532120 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.532080 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.532317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.532133 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6mjn\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-kube-api-access-m6mjn\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:09.532317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:09.532152 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec3c61f-ca5f-4bb5-af09-8885570c37e8-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:10.011344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.011311 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44251e9-40b9-4e52-8ca7-14cbbb156036" path="/var/lib/kubelet/pods/e44251e9-40b9-4e52-8ca7-14cbbb156036/volumes" Nov 03 16:38:10.336026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.335934 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" Nov 03 16:38:10.336026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.336007 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59" event={"ID":"1ec3c61f-ca5f-4bb5-af09-8885570c37e8","Type":"ContainerDied","Data":"b7c81ea42ef25498530ff10e297c1816b90a1762c5bcb9f77ec773807686615a"} Nov 03 16:38:10.336631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.336068 5946 scope.go:122] "RemoveContainer" containerID="9f598bc7fe8791d40da0306afc9a9d7a210b5e5b4f47e2da6f89965c45584d1d" Nov 03 16:38:10.435618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.435589 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:10.562244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.562190 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59"] Nov 03 16:38:10.612942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.612867 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-n7r59"] Nov 03 16:38:10.658570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.658539 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:10.700950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.700924 5946 scope.go:122] "RemoveContainer" containerID="44b922d79015544449a9573ff78e66947ab3b876e6514fc4d4480ccd07e2dfa9" Nov 03 16:38:10.712078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.712050 5946 scope.go:122] "RemoveContainer" containerID="3b0c20887077d13e967aaa9c7d8578605a9d68f05faccc623ebde1a6dccc0ca1" Nov 03 16:38:10.725128 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.725103 5946 scope.go:122] "RemoveContainer" containerID="6c50b8c89f2c34e6e825a193b77fcbf3514fb7fedd314f51aad19cf610e965ef" Nov 03 16:38:10.735925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.735907 5946 scope.go:122] "RemoveContainer" containerID="6c07fb5a361659bcd39ad7df278b102d9827b196583533e3c8b0f26c164f6758" Nov 03 16:38:10.748999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.748981 5946 scope.go:122] "RemoveContainer" containerID="bc21b1596942e0b6a037a92808a8e7e2225b7e92772ffea1edccf4a395e4b879" Nov 03 16:38:10.760998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.760976 5946 scope.go:122] "RemoveContainer" containerID="d0a7cdd6e51957d3c1a265ad23101bddce9b779547e9269a87e0b570e9784541" Nov 03 16:38:10.808683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.808644 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q"] Nov 03 16:38:10.808952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.808925 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" containerID="cri-o://0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781" gracePeriod=30 Nov 03 16:38:10.868175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:10.868112 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:11.362429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:11.362400 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:11.566605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:11.564146 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:11.827500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:11.827464 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:12.005739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:12.005707 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" path="/var/lib/kubelet/pods/1ec3c61f-ca5f-4bb5-af09-8885570c37e8/volumes" Nov 03 16:38:12.421367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:12.421334 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:12.588042 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:38:12.587996 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:38:14.068302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.068265 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc"] Nov 03 16:38:14.074812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.074777 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed516fd-gssfc"] Nov 03 16:38:14.188988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.188956 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:38:14.802089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.802008 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq"] Nov 03 16:38:14.802769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.802750 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" Nov 03 16:38:14.802835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.802769 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" Nov 03 16:38:14.802895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.802886 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec3c61f-ca5f-4bb5-af09-8885570c37e8" containerName="istio-proxy" Nov 03 16:38:14.808596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.808577 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.847349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.847321 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqkj\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-kube-api-access-fjqkj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.847507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.847451 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.847507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.847486 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.847594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.847550 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.847594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.847580 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.847663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.847626 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f56784-9ff3-4b95-866a-e9b309101063-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.847842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.847822 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.868353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.868327 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 16:38:14.868353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.868332 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 16:38:14.868562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.868378 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 16:38:14.868562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.868418 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-57mjv\"" Nov 03 16:38:14.911622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.911590 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq"] Nov 03 16:38:14.949901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.949858 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.949901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.949904 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqkj\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-kube-api-access-fjqkj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.950161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.950008 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.950161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.950026 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.950161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.950082 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.950161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.950104 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.950161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.950144 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f56784-9ff3-4b95-866a-e9b309101063-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.951110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.951051 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.952962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.952915 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f56784-9ff3-4b95-866a-e9b309101063-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.953303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.953281 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.953417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.953402 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:14.953656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:14.953634 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:15.014830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:15.014800 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqkj\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-kube-api-access-fjqkj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:15.015703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:15.015679 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b94nq\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:15.122596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:15.122501 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:15.584081 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:38:15.584036 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f56784_9ff3_4b95_866a_e9b309101063.slice/crio-7d0d84792dc5ca5bd9b6c7b334dfea76d73eed51dd9144e0b8432d38fd2aa386 WatchSource:0}: Error finding container 7d0d84792dc5ca5bd9b6c7b334dfea76d73eed51dd9144e0b8432d38fd2aa386: Status 404 returned error can't find the container with id 7d0d84792dc5ca5bd9b6c7b334dfea76d73eed51dd9144e0b8432d38fd2aa386 Nov 03 16:38:15.586370 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:15.586347 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq"] Nov 03 16:38:16.021771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.021732 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04904dd-d6d5-49b1-b2d5-39e41bc10fc3" path="/var/lib/kubelet/pods/e04904dd-d6d5-49b1-b2d5-39e41bc10fc3/volumes" Nov 03 16:38:16.078345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.078317 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp"] Nov 03 16:38:16.096825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.096801 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:38:16.128845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.128816 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0c4eab-lg6fp"] Nov 03 16:38:16.175701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.175671 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-envoy\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.175848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.175707 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-credential-socket\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.175848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.175749 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-kube-api-access-r87m2\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-kube-api-access-r87m2\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.175848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.175782 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-istio-token\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-istio-token\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.175999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.175973 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-credential-socket" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:16.176040 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.175999 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-data\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.176087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176044 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-certs\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.176326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176295 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-data" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:16.176326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176313 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f24c959b-1822-45e4-a423-942e8af1fb26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c959b-1822-45e4-a423-942e8af1fb26-istio-podinfo\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.176326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176315 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-certs" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:16.176573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176348 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24c959b-1822-45e4-a423-942e8af1fb26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c959b-1822-45e4-a423-942e8af1fb26-istiod-ca-cert\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.176573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176393 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-socket\") pod \"f24c959b-1822-45e4-a423-942e8af1fb26\" (UID: \"f24c959b-1822-45e4-a423-942e8af1fb26\") " Nov 03 16:38:16.176836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176803 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-socket" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:16.176931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.176906 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24c959b-1822-45e4-a423-942e8af1fb26-istiod-ca-cert" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:16.177286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.177265 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.177367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.177290 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.177367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.177304 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.177367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.177318 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c959b-1822-45e4-a423-942e8af1fb26-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.177367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.177331 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.178484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.178434 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-envoy" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:16.178567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.178481 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-kube-api-access-r87m2" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "kube-api-access-r87m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:16.178730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.178716 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-istio-token" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:16.178849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.178827 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f24c959b-1822-45e4-a423-942e8af1fb26-istio-podinfo" pod "f24c959b-1822-45e4-a423-942e8af1fb26" (UID: "f24c959b-1822-45e4-a423-942e8af1fb26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:38:16.279079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.279043 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c959b-1822-45e4-a423-942e8af1fb26-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.279079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.279075 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c959b-1822-45e4-a423-942e8af1fb26-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.279079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.279085 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r87m2\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-kube-api-access-r87m2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.279381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.279095 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c959b-1822-45e4-a423-942e8af1fb26-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:16.392440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.392406 5946 generic.go:356] "Generic (PLEG): container finished" podID="f24c959b-1822-45e4-a423-942e8af1fb26" containerID="0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781" exitCode=0 Nov 03 16:38:16.392605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.392464 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" Nov 03 16:38:16.392605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.392491 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" event={"ID":"f24c959b-1822-45e4-a423-942e8af1fb26","Type":"ContainerDied","Data":"0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781"} Nov 03 16:38:16.392605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.392545 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q" event={"ID":"f24c959b-1822-45e4-a423-942e8af1fb26","Type":"ContainerDied","Data":"3de3cfeeb63776aa025476c4a03ad9f840b46d398312c3a9e588d543e46289fc"} Nov 03 16:38:16.392605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.392574 5946 scope.go:122] "RemoveContainer" containerID="0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781" Nov 03 16:38:16.394420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.394398 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" event={"ID":"d4f56784-9ff3-4b95-866a-e9b309101063","Type":"ContainerStarted","Data":"7d0d84792dc5ca5bd9b6c7b334dfea76d73eed51dd9144e0b8432d38fd2aa386"} Nov 03 16:38:16.404884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.404861 5946 scope.go:122] "RemoveContainer" containerID="d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be" Nov 03 16:38:16.415856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.415837 5946 scope.go:122] "RemoveContainer" containerID="0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781" Nov 03 16:38:16.416127 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:38:16.416108 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781\": container with ID starting with 0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781 not found: ID does not exist" containerID="0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781" Nov 03 16:38:16.416194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.416142 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781"} err="failed to get container status \"0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781\": rpc error: code = NotFound desc = could not find container \"0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781\": container with ID starting with 0b8522a01f0d214226441c633529c019002aa3dffc73622ecc2945ec0a0b6781 not found: ID does not exist" Nov 03 16:38:16.416194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.416181 5946 scope.go:122] "RemoveContainer" containerID="d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be" Nov 03 16:38:16.416501 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:38:16.416462 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be\": container with ID starting with d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be not found: ID does not exist" containerID="d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be" Nov 03 16:38:16.416501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.416489 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be"} err="failed to get container status \"d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be\": rpc error: code = NotFound desc = could not find container \"d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be\": container with ID starting with d55699d053c19de7ffe5a44840cc1d5b6ca85c49faee360b8dca43190f2da0be not found: ID does not exist" Nov 03 16:38:16.567070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.567041 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q"] Nov 03 16:38:16.572839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:16.572812 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06f127ec-openshift-default-696574f8f-8ts7q"] Nov 03 16:38:17.387888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:17.387844 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:18.006598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:18.006570 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f263ec-23f8-4cb9-a4f0-55969ca04b6b" path="/var/lib/kubelet/pods/c6f263ec-23f8-4cb9-a4f0-55969ca04b6b/volumes" Nov 03 16:38:18.007093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:18.007080 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" path="/var/lib/kubelet/pods/f24c959b-1822-45e4-a423-942e8af1fb26/volumes" Nov 03 16:38:19.541690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:19.541659 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:19.739276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:19.739183 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:20.216889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:20.216852 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:20.454055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:20.454024 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:21.626694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:21.626659 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:21.871402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:21.871370 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:22.062975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:22.062941 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r"] Nov 03 16:38:22.118692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:22.118649 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:22.126966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:22.126931 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7834cd86-4bt8r"] Nov 03 16:38:24.008778 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:24.008742 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62f356e-67cf-4a8c-b246-31996edeac03" path="/var/lib/kubelet/pods/e62f356e-67cf-4a8c-b246-31996edeac03/volumes" Nov 03 16:38:25.065236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:25.065166 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm"] Nov 03 16:38:25.113972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:25.113937 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62592796-c5qnm"] Nov 03 16:38:25.530282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:25.530250 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:38:25.530400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:25.530311 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:38:26.007517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:26.007485 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa658d9-75a6-4e03-aa47-7d7266dd19eb" path="/var/lib/kubelet/pods/0fa658d9-75a6-4e03-aa47-7d7266dd19eb/volumes" Nov 03 16:38:26.499746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:26.499702 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" event={"ID":"d4f56784-9ff3-4b95-866a-e9b309101063","Type":"ContainerStarted","Data":"48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf"} Nov 03 16:38:26.500186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:26.499779 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:26.502073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:26.502047 5946 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b94nq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:38:26.502199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:26.502095 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" podUID="d4f56784-9ff3-4b95-866a-e9b309101063" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:38:26.714122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:26.714064 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" podStartSLOduration=2.771460593 podStartE2EDuration="12.714043843s" totalImagesPullingTime="9.94258325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:26.675621733 +0000 UTC m=+1576.796674556" watchObservedRunningTime="2026-11-03 16:38:26.714043843 +0000 UTC m=+1576.835096666" Nov 03 16:38:26.974081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:26.973949 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:27.071169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:27.071132 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb"] Nov 03 16:38:27.077194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:27.077162 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b4cecf-69bgb"] Nov 03 16:38:27.509717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:27.509684 5946 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b94nq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:38:27.510121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:27.509754 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" podUID="d4f56784-9ff3-4b95-866a-e9b309101063" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:38:28.006418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.006387 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd837f4-986a-4b71-82f7-d58b77d78362" path="/var/lib/kubelet/pods/5dd837f4-986a-4b71-82f7-d58b77d78362/volumes" Nov 03 16:38:28.825028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.824993 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m"] Nov 03 16:38:28.825752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.825734 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" Nov 03 16:38:28.825801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.825753 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" Nov 03 16:38:28.825801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.825766 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" Nov 03 16:38:28.825801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.825771 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" Nov 03 16:38:28.825905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.825896 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" Nov 03 16:38:28.825940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.825908 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24c959b-1822-45e4-a423-942e8af1fb26" containerName="istio-proxy" Nov 03 16:38:28.830819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.830799 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:28.889291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.889259 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m"] Nov 03 16:38:28.890592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.890559 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:38:28.890745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.890606 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:38:28.890745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.890622 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:38:28.930458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.930418 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:38:28.991612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.991581 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:28.991840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.991820 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85l6v\" (UniqueName: \"kubernetes.io/projected/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-kube-api-access-85l6v\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:28.991890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:28.991872 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.093772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.093684 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85l6v\" (UniqueName: \"kubernetes.io/projected/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-kube-api-access-85l6v\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.093772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.093740 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.093976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.093837 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.094550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.094522 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.096379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.096354 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.154457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.154427 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85l6v\" (UniqueName: \"kubernetes.io/projected/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-kube-api-access-85l6v\") pod \"maas-api-revoke-sub-ab49f2ef-ndn9m\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.445914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.445827 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:29.835570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:29.835539 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m"] Nov 03 16:38:29.836775 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:38:29.836747 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2650b82e_ba36_459b_8f2d_3d6b1d5854d8.slice/crio-5e8bbbe08493a662a112011cc5fa06b7669c50f9b693b70ae521dcc513d572cb WatchSource:0}: Error finding container 5e8bbbe08493a662a112011cc5fa06b7669c50f9b693b70ae521dcc513d572cb: Status 404 returned error can't find the container with id 5e8bbbe08493a662a112011cc5fa06b7669c50f9b693b70ae521dcc513d572cb Nov 03 16:38:30.510084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:30.510056 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:38:30.554364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:30.554316 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" event={"ID":"2650b82e-ba36-459b-8f2d-3d6b1d5854d8","Type":"ContainerStarted","Data":"c1e0b3f1ba8e9ca5304c0719594173d3fc3681b2f4a4a140649ffd79c9049340"} Nov 03 16:38:30.554560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:30.554425 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" event={"ID":"2650b82e-ba36-459b-8f2d-3d6b1d5854d8","Type":"ContainerStarted","Data":"5e8bbbe08493a662a112011cc5fa06b7669c50f9b693b70ae521dcc513d572cb"} Nov 03 16:38:30.856013 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:30.855908 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" podStartSLOduration=2.855892358 podStartE2EDuration="2.855892358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:30.854500833 +0000 UTC m=+1580.975553655" watchObservedRunningTime="2026-11-03 16:38:30.855892358 +0000 UTC m=+1580.976945181" Nov 03 16:38:31.567463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:31.567425 5946 generic.go:356] "Generic (PLEG): container finished" podID="2650b82e-ba36-459b-8f2d-3d6b1d5854d8" containerID="c1e0b3f1ba8e9ca5304c0719594173d3fc3681b2f4a4a140649ffd79c9049340" exitCode=0 Nov 03 16:38:31.567672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:31.567501 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" event={"ID":"2650b82e-ba36-459b-8f2d-3d6b1d5854d8","Type":"ContainerDied","Data":"c1e0b3f1ba8e9ca5304c0719594173d3fc3681b2f4a4a140649ffd79c9049340"} Nov 03 16:38:31.702074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:31.702033 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:31.912273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:31.912160 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:32.450267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:32.447574 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:32.847000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:32.846833 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:32.927867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:32.927837 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:33.036136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.036094 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-xks-ca\") pod \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " Nov 03 16:38:33.036394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.036343 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-openshift-ca\") pod \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " Nov 03 16:38:33.036467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.036425 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-kube-api-access-85l6v\" (UniqueName: \"kubernetes.io/projected/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-kube-api-access-85l6v\") pod \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\" (UID: \"2650b82e-ba36-459b-8f2d-3d6b1d5854d8\") " Nov 03 16:38:33.036777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.036753 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-openshift-ca" pod "2650b82e-ba36-459b-8f2d-3d6b1d5854d8" (UID: "2650b82e-ba36-459b-8f2d-3d6b1d5854d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:33.037292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.037261 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:33.039473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.039445 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-xks-ca" pod "2650b82e-ba36-459b-8f2d-3d6b1d5854d8" (UID: "2650b82e-ba36-459b-8f2d-3d6b1d5854d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:33.047418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.047390 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-kube-api-access-85l6v" pod "2650b82e-ba36-459b-8f2d-3d6b1d5854d8" (UID: "2650b82e-ba36-459b-8f2d-3d6b1d5854d8"). InnerVolumeSpecName "kube-api-access-85l6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:33.139639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.139388 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:33.139639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.139422 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85l6v\" (UniqueName: \"kubernetes.io/projected/2650b82e-ba36-459b-8f2d-3d6b1d5854d8-kube-api-access-85l6v\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:33.386977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.386948 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:33.590355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.590323 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" event={"ID":"2650b82e-ba36-459b-8f2d-3d6b1d5854d8","Type":"ContainerDied","Data":"5e8bbbe08493a662a112011cc5fa06b7669c50f9b693b70ae521dcc513d572cb"} Nov 03 16:38:33.590545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.590371 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e8bbbe08493a662a112011cc5fa06b7669c50f9b693b70ae521dcc513d572cb" Nov 03 16:38:33.590545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:33.590376 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m" Nov 03 16:38:34.065331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:34.065298 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk"] Nov 03 16:38:34.116866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:34.116832 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167f28b4-92pnk"] Nov 03 16:38:34.877099 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:34.877068 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:36.222232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:36.217478 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:36.276014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:36.275492 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4589e023-68f5-417e-a830-a7324b300e7a" path="/var/lib/kubelet/pods/4589e023-68f5-417e-a830-a7324b300e7a/volumes" Nov 03 16:38:36.445756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:36.445714 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:36.867648 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:36.867611 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:37.083387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:37.083354 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:37.286771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:37.286733 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:37.985323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:37.985294 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:38.173956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:38.173925 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:38.566379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:38.566347 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:38.778009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:38.777976 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:38.988226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:38.985728 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:39.617250 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:39.617179 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:41.311456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:41.311425 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:41.507575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:41.507547 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:42.104994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:42.104957 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:42.588306 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:38:42.588266 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:38:43.281685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.281651 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq"] Nov 03 16:38:43.282343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.282328 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2650b82e-ba36-459b-8f2d-3d6b1d5854d8" containerName="revoke-keys" Nov 03 16:38:43.282397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.282345 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2650b82e-ba36-459b-8f2d-3d6b1d5854d8" containerName="revoke-keys" Nov 03 16:38:43.282487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.282478 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="2650b82e-ba36-459b-8f2d-3d6b1d5854d8" containerName="revoke-keys" Nov 03 16:38:43.287373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.287352 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.345749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.345719 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq"] Nov 03 16:38:43.379502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.379469 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhqkk\" (UniqueName: \"kubernetes.io/projected/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-kube-api-access-qhqkk\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.379698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.379625 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.379808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.379779 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.391550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.391516 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:38:43.391550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.391543 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:38:43.391805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.391542 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:38:43.393831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.393807 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:38:43.481760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.481725 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhqkk\" (UniqueName: \"kubernetes.io/projected/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-kube-api-access-qhqkk\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.481954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.481901 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.482047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.482022 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.482837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.482810 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.484569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.484543 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.549905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.549832 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhqkk\" (UniqueName: \"kubernetes.io/projected/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-kube-api-access-qhqkk\") pod \"maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.601835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.601804 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:43.706553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.706494 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:43.906896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:43.906794 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:44.018845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:44.018814 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq"] Nov 03 16:38:44.019818 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:38:44.019793 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47f2889_9b30_4ca2_95e8_fb4e2e1f0764.slice/crio-81c4507acadda683293909e567fc9bfe74ebd72a0cb85f72ec8d04a34f3f8f5c WatchSource:0}: Error finding container 81c4507acadda683293909e567fc9bfe74ebd72a0cb85f72ec8d04a34f3f8f5c: Status 404 returned error can't find the container with id 81c4507acadda683293909e567fc9bfe74ebd72a0cb85f72ec8d04a34f3f8f5c Nov 03 16:38:44.702356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:44.702257 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" event={"ID":"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764","Type":"ContainerStarted","Data":"d432572032767c454740f34cefd05e99063839d0b878e3ec1a82edc66e6f9d57"} Nov 03 16:38:44.702356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:44.702328 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" event={"ID":"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764","Type":"ContainerStarted","Data":"81c4507acadda683293909e567fc9bfe74ebd72a0cb85f72ec8d04a34f3f8f5c"} Nov 03 16:38:44.827217 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:44.827161 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" podStartSLOduration=1.82714725 podStartE2EDuration="1.82714725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:44.8243781 +0000 UTC m=+1594.945430920" watchObservedRunningTime="2026-11-03 16:38:44.82714725 +0000 UTC m=+1594.948200071" Nov 03 16:38:45.713644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:45.713607 5946 generic.go:356] "Generic (PLEG): container finished" podID="b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" containerID="d432572032767c454740f34cefd05e99063839d0b878e3ec1a82edc66e6f9d57" exitCode=0 Nov 03 16:38:45.714078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:45.713697 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" event={"ID":"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764","Type":"ContainerDied","Data":"d432572032767c454740f34cefd05e99063839d0b878e3ec1a82edc66e6f9d57"} Nov 03 16:38:46.901748 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.901721 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:46.960676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.960629 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-kube-api-access-qhqkk\" (UniqueName: \"kubernetes.io/projected/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-kube-api-access-qhqkk\") pod \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " Nov 03 16:38:46.960847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.960703 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-openshift-ca\") pod \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " Nov 03 16:38:46.960997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.960980 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-xks-ca\") pod \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\" (UID: \"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764\") " Nov 03 16:38:46.961119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.961087 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-openshift-ca" pod "b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" (UID: "b47f2889-9b30-4ca2-95e8-fb4e2e1f0764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:46.961731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.961706 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:46.963127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.963102 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-kube-api-access-qhqkk" pod "b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" (UID: "b47f2889-9b30-4ca2-95e8-fb4e2e1f0764"). InnerVolumeSpecName "kube-api-access-qhqkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:46.963377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:46.963355 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-xks-ca" pod "b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" (UID: "b47f2889-9b30-4ca2-95e8-fb4e2e1f0764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:47.064437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:47.064403 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:47.064794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:47.064769 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhqkk\" (UniqueName: \"kubernetes.io/projected/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764-kube-api-access-qhqkk\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:47.289652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:47.289618 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:47.484971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:47.484937 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:47.737287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:47.737260 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" Nov 03 16:38:47.737457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:47.737298 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq" event={"ID":"b47f2889-9b30-4ca2-95e8-fb4e2e1f0764","Type":"ContainerDied","Data":"81c4507acadda683293909e567fc9bfe74ebd72a0cb85f72ec8d04a34f3f8f5c"} Nov 03 16:38:47.737457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:47.737344 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81c4507acadda683293909e567fc9bfe74ebd72a0cb85f72ec8d04a34f3f8f5c" Nov 03 16:38:51.981639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:51.981233 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:52.559611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:52.559574 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:52.772755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:52.772717 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:53.647392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.647341 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n"] Nov 03 16:38:53.649231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.649164 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" containerName="revoke-keys" Nov 03 16:38:53.649423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.649405 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" containerName="revoke-keys" Nov 03 16:38:53.649737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.649705 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" containerName="revoke-keys" Nov 03 16:38:53.728230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.727400 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:53.762519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.744441 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n"] Nov 03 16:38:53.804306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.788416 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4w65\" (UniqueName: \"kubernetes.io/projected/64ad51b6-004e-458b-8851-764b3aae7795-kube-api-access-r4w65\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:53.843532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.804825 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:53.843777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.802569 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:38:53.843900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.802504 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:38:53.843983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.843914 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:53.862775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.862743 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:38:53.863152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.863129 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:38:53.979310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.972744 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:54.099349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:54.099312 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:54.102343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:54.099807 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4w65\" (UniqueName: \"kubernetes.io/projected/64ad51b6-004e-458b-8851-764b3aae7795-kube-api-access-r4w65\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:54.104878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:53.995091 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:54.104878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:54.104277 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:54.197286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:54.179891 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4w65\" (UniqueName: \"kubernetes.io/projected/64ad51b6-004e-458b-8851-764b3aae7795-kube-api-access-r4w65\") pod \"maas-api-revoke-sub-c1b0ac4d-ln49n\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:54.378232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:54.377528 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:54.995127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:54.995088 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n"] Nov 03 16:38:55.072407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:55.072331 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8"] Nov 03 16:38:55.078975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:55.078942 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2725e103-ssvl8"] Nov 03 16:38:55.114200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:55.114161 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" event={"ID":"64ad51b6-004e-458b-8851-764b3aae7795","Type":"ContainerStarted","Data":"e0168d5f001910d93c393587628c2da185604d6f1a8b4ee752f90705cbedc5b0"} Nov 03 16:38:56.005793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:56.005762 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d9df82-06d5-4ef7-93d7-073786ca83aa" path="/var/lib/kubelet/pods/b5d9df82-06d5-4ef7-93d7-073786ca83aa/volumes" Nov 03 16:38:56.124744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:56.124709 5946 generic.go:356] "Generic (PLEG): container finished" podID="64ad51b6-004e-458b-8851-764b3aae7795" containerID="a2c610d9a71ceabfde905d3ebcd31c52b31c24dd35d4b4bee560bfab58ea4ab5" exitCode=0 Nov 03 16:38:56.124917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:56.124796 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" event={"ID":"64ad51b6-004e-458b-8851-764b3aae7795","Type":"ContainerDied","Data":"a2c610d9a71ceabfde905d3ebcd31c52b31c24dd35d4b4bee560bfab58ea4ab5"} Nov 03 16:38:56.921098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:56.921060 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:57.126469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.126435 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:57.322843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.322793 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:57.336716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.336688 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:57.446836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.446803 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-xks-ca\") pod \"64ad51b6-004e-458b-8851-764b3aae7795\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " Nov 03 16:38:57.447014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.446949 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-openshift-ca\") pod \"64ad51b6-004e-458b-8851-764b3aae7795\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " Nov 03 16:38:57.447014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.446994 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ad51b6-004e-458b-8851-764b3aae7795-kube-api-access-r4w65\" (UniqueName: \"kubernetes.io/projected/64ad51b6-004e-458b-8851-764b3aae7795-kube-api-access-r4w65\") pod \"64ad51b6-004e-458b-8851-764b3aae7795\" (UID: \"64ad51b6-004e-458b-8851-764b3aae7795\") " Nov 03 16:38:57.447394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.447368 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-openshift-ca" pod "64ad51b6-004e-458b-8851-764b3aae7795" (UID: "64ad51b6-004e-458b-8851-764b3aae7795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:57.447907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.447868 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:57.449435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.449402 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ad51b6-004e-458b-8851-764b3aae7795-kube-api-access-r4w65" pod "64ad51b6-004e-458b-8851-764b3aae7795" (UID: "64ad51b6-004e-458b-8851-764b3aae7795"). InnerVolumeSpecName "kube-api-access-r4w65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:57.449597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.449579 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-xks-ca" pod "64ad51b6-004e-458b-8851-764b3aae7795" (UID: "64ad51b6-004e-458b-8851-764b3aae7795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:57.549913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.549443 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad51b6-004e-458b-8851-764b3aae7795-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:57.549913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:57.549506 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4w65\" (UniqueName: \"kubernetes.io/projected/64ad51b6-004e-458b-8851-764b3aae7795-kube-api-access-r4w65\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:58.143931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:58.143900 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" Nov 03 16:38:58.144348 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:58.143932 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n" event={"ID":"64ad51b6-004e-458b-8851-764b3aae7795","Type":"ContainerDied","Data":"e0168d5f001910d93c393587628c2da185604d6f1a8b4ee752f90705cbedc5b0"} Nov 03 16:38:58.144348 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:58.143983 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0168d5f001910d93c393587628c2da185604d6f1a8b4ee752f90705cbedc5b0" Nov 03 16:38:59.361200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:59.361167 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:59.549359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:38:59.549328 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:01.912631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:01.912591 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:04.525374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:04.525304 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:06.763319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:06.763257 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:07.065415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:07.065326 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26"] Nov 03 16:39:07.116833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:07.116798 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f1fd9e-64h26"] Nov 03 16:39:08.007172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:08.007140 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183e7f2b-d882-498a-a85a-2a2a0a88025f" path="/var/lib/kubelet/pods/183e7f2b-d882-498a-a85a-2a2a0a88025f/volumes" Nov 03 16:39:10.877256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.877221 5946 scope.go:122] "RemoveContainer" containerID="8155552fccb170afce2b8d9007d6dfc2b8b2ccac70eb8f5b2cf7646c25c385ef" Nov 03 16:39:10.888510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.888480 5946 scope.go:122] "RemoveContainer" containerID="188ae47ac10c5af56a66f83b0b5c402fce5c1e3713fa3bcbc48f8978bfdaaec5" Nov 03 16:39:10.900524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.900500 5946 scope.go:122] "RemoveContainer" containerID="889b630ee8731210359001627c1189f8725c523eef356c7b0271d692dbb9cfae" Nov 03 16:39:10.911256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.911233 5946 scope.go:122] "RemoveContainer" containerID="082005ff86bd8e48af417ad87e360479176b01e592ecdbac71bfb3d4d23b2167" Nov 03 16:39:10.922702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.922681 5946 scope.go:122] "RemoveContainer" containerID="21e19d64a9f36e40d981c95dbeb34b988f19df9ff704e40ed0265eddbc526a96" Nov 03 16:39:10.932292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.932268 5946 scope.go:122] "RemoveContainer" containerID="2f04c33b03d0d9ece2c0cba008b3ea910abb47add5f97755fe6a0ef606ed38fb" Nov 03 16:39:10.941894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.941869 5946 scope.go:122] "RemoveContainer" containerID="6058cdab0483d18514bd88f1e99510f6451f19ec2788fbbd836a776ac120d20a" Nov 03 16:39:10.951379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.951362 5946 scope.go:122] "RemoveContainer" containerID="53a9a3e449405c4b5651999f1030306d9965d7f6bdcfe938e02248fcd26789c7" Nov 03 16:39:10.960929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.960910 5946 scope.go:122] "RemoveContainer" containerID="46f7e97d265f697d823bd21ff8c052a4fc0ffd929c95e69df5f45e0374a732d8" Nov 03 16:39:10.970085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.970065 5946 scope.go:122] "RemoveContainer" containerID="6aa9ada5017d8a9bfc106e4c58b27145ee2307ad799f10682a8f17d1ea49d71b" Nov 03 16:39:10.981899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.981880 5946 scope.go:122] "RemoveContainer" containerID="bf90f5623c08d09d8d1b06d5f1228ab0818f708366489f3285c46eb1ab722c62" Nov 03 16:39:10.992549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:10.992532 5946 scope.go:122] "RemoveContainer" containerID="d4203528f3ac050ac62177b7e08d27fa07fcdb43753999903f6c1191bf5ac5ad" Nov 03 16:39:11.001840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:11.001821 5946 scope.go:122] "RemoveContainer" containerID="412659bf81e5a849139342c6dc50d0e2bc1c05cda4a2410df5ecee4e0f9578c7" Nov 03 16:39:11.010719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:11.010704 5946 scope.go:122] "RemoveContainer" containerID="846c25e022bc4c7c500fceef22ae39193c831933fbac235b1018b5820bb82ef6" Nov 03 16:39:11.565902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:11.565848 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:12.363350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:12.363320 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:12.587569 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:12.587530 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:39:16.290258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:16.282922 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:18.165164 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.165131 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c"] Nov 03 16:39:18.165718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.165399 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" containerID="cri-o://ec2d4d85b8cf86717213016776b05f9f65c1c63225e2cbdf4bac0d60d6151b3e" gracePeriod=30 Nov 03 16:39:18.217749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.217718 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82"] Nov 03 16:39:18.217948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.217929 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" containerID="cri-o://9425dcc24f131f0a0aecd8d66d5b0348ee82643ce08e817bfaaaa70df795f301" gracePeriod=30 Nov 03 16:39:18.222937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.222908 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr"] Nov 03 16:39:18.223167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.223126 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" containerID="cri-o://2b0cfcf3ab7da0b1497b29efd71ef4fc56bdb88610171944d8826346491e15bb" gracePeriod=30 Nov 03 16:39:18.510315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.510278 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr"] Nov 03 16:39:18.510771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:18.510732 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" containerID="cri-o://d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9" gracePeriod=30 Nov 03 16:39:19.526127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:19.526092 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt"] Nov 03 16:39:19.526569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:19.526347 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" containerID="cri-o://f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79" gracePeriod=30 Nov 03 16:39:19.869158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:19.869059 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9"] Nov 03 16:39:19.869368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:19.869336 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" containerID="cri-o://d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53" gracePeriod=30 Nov 03 16:39:21.035369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:21.035331 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:22.929474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:22.929442 5946 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:39:22.929894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:22.929517 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:39:23.385369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.385328 5946 generic.go:356] "Generic (PLEG): container finished" podID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerID="9425dcc24f131f0a0aecd8d66d5b0348ee82643ce08e817bfaaaa70df795f301" exitCode=0 Nov 03 16:39:23.385589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.385392 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" event={"ID":"e3431a37-3f66-47e5-bcff-61ba7ebeeb90","Type":"ContainerDied","Data":"9425dcc24f131f0a0aecd8d66d5b0348ee82643ce08e817bfaaaa70df795f301"} Nov 03 16:39:23.387100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.387071 5946 generic.go:356] "Generic (PLEG): container finished" podID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerID="2b0cfcf3ab7da0b1497b29efd71ef4fc56bdb88610171944d8826346491e15bb" exitCode=0 Nov 03 16:39:23.387252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.387192 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" event={"ID":"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430","Type":"ContainerDied","Data":"2b0cfcf3ab7da0b1497b29efd71ef4fc56bdb88610171944d8826346491e15bb"} Nov 03 16:39:23.390105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.390081 5946 generic.go:356] "Generic (PLEG): container finished" podID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerID="ec2d4d85b8cf86717213016776b05f9f65c1c63225e2cbdf4bac0d60d6151b3e" exitCode=0 Nov 03 16:39:23.390314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.390291 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" event={"ID":"e589bc27-2ffb-4627-8dc4-2ed8a606e716","Type":"ContainerDied","Data":"ec2d4d85b8cf86717213016776b05f9f65c1c63225e2cbdf4bac0d60d6151b3e"} Nov 03 16:39:23.631751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.631724 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:39:23.635654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.635632 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:39:23.639556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.639538 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:39:23.785456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785375 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-kube-api-access-s7ccd\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-kube-api-access-s7ccd\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.785456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785417 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-socket\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.785456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785440 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-socket\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.785724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785578 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-kube-api-access-c6lff\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-kube-api-access-c6lff\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.785724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785628 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-credential-socket\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.785724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785654 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-envoy\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.785724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785705 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-envoy\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.785953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785750 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-data\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.785953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785793 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-credential-socket\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.785953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785795 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-socket" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.785953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785826 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-data\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.785953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785904 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istiod-ca-cert\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.785953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785950 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-token\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785962 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-credential-socket" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785984 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-certs\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786036 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-data" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786073 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-envoy\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.785783 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-socket" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786146 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-credential-socket\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786176 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-certs" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786176 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-podinfo\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.786285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786279 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-data" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786283 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-socket\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786372 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istiod-ca-cert\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786456 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-podinfo\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786466 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-socket" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786540 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-kube-api-access-zn5h6\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-kube-api-access-zn5h6\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786640 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-token\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-token\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786702 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-podinfo\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.786745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786749 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-data\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786776 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istiod-ca-cert\") pod \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\" (UID: \"e589bc27-2ffb-4627-8dc4-2ed8a606e716\") " Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786798 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istiod-ca-cert" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786814 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-credential-socket" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786824 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-token\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-token\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786853 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-credential-socket" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786884 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-certs\") pod \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\" (UID: \"e3431a37-3f66-47e5-bcff-61ba7ebeeb90\") " Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.786922 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-certs\") pod \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\" (UID: \"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430\") " Nov 03 16:39:23.787291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.787125 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-data" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.787937 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.787965 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.787980 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.787995 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788009 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788024 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788040 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788054 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788067 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788080 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788094 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.788892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788357 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-certs" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.788892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.788718 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-kube-api-access-c6lff" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "kube-api-access-c6lff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:23.789310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.789284 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istiod-ca-cert" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:23.789399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.789363 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-token" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:23.789541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.789429 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istiod-ca-cert" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:23.789541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.789477 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-certs" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.789969 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.789908 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-kube-api-access-s7ccd" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "kube-api-access-s7ccd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:23.790376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.790167 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-podinfo" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:23.790376 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.790326 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-envoy" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.790669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.790511 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-envoy" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.790669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.790525 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-envoy" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:23.791688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.791665 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-podinfo" pod "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" (UID: "7b04a7e3-f1ae-4b8a-863d-c821f4eeb430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:23.791820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.791801 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-podinfo" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:23.791889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.791830 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-token" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:23.792166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.792151 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-token" pod "e589bc27-2ffb-4627-8dc4-2ed8a606e716" (UID: "e589bc27-2ffb-4627-8dc4-2ed8a606e716"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:23.792374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.792352 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-kube-api-access-zn5h6" pod "e3431a37-3f66-47e5-bcff-61ba7ebeeb90" (UID: "e3431a37-3f66-47e5-bcff-61ba7ebeeb90"). InnerVolumeSpecName "kube-api-access-zn5h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:23.890005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.889972 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lff\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-kube-api-access-c6lff\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890000 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890010 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890017 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890026 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890036 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890045 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890053 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890062 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn5h6\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-kube-api-access-zn5h6\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890071 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890079 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890089 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e589bc27-2ffb-4627-8dc4-2ed8a606e716-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890098 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890105 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3431a37-3f66-47e5-bcff-61ba7ebeeb90-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890113 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:23.890273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:23.890121 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ccd\" (UniqueName: \"kubernetes.io/projected/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430-kube-api-access-s7ccd\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.265047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.265022 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:39:24.402315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.402190 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" Nov 03 16:39:24.402315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.402261 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr" event={"ID":"7b04a7e3-f1ae-4b8a-863d-c821f4eeb430","Type":"ContainerDied","Data":"2bda2cb39b70f8e9fdb79256686bbf85cdf36af210661676ef5e29897751ac51"} Nov 03 16:39:24.402539 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.402322 5946 scope.go:122] "RemoveContainer" containerID="2b0cfcf3ab7da0b1497b29efd71ef4fc56bdb88610171944d8826346491e15bb" Nov 03 16:39:24.403939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.403603 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-kube-api-access-flk6p\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-kube-api-access-flk6p\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.403939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.403702 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-envoy\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.403939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.403738 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-certs\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.403939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.403874 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-socket\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.404177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.403981 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3176a5ac-4339-4bde-b9e4-db18492adf2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3176a5ac-4339-4bde-b9e4-db18492adf2e-istiod-ca-cert\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.404177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.404167 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-podinfo\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.404355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.404266 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-data\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.404355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.404324 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-credential-socket\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.404582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.404563 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-credential-socket" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:24.404676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.404584 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-socket" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:24.404676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.404629 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-token\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-token\") pod \"3176a5ac-4339-4bde-b9e4-db18492adf2e\" (UID: \"3176a5ac-4339-4bde-b9e4-db18492adf2e\") " Nov 03 16:39:24.405025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.404960 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-data" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:24.405234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.405161 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3176a5ac-4339-4bde-b9e4-db18492adf2e-istiod-ca-cert" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:24.405564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.405531 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" Nov 03 16:39:24.405692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.405535 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c" event={"ID":"e589bc27-2ffb-4627-8dc4-2ed8a606e716","Type":"ContainerDied","Data":"a50331422ed83e75f872948a2bb66a6b4ec903e31457561aaed975d11439a892"} Nov 03 16:39:24.406678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.406654 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.406797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.406684 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3176a5ac-4339-4bde-b9e4-db18492adf2e-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.406797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.406737 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.406797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.406755 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.406797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.406765 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-certs" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:24.407078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.406963 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-kube-api-access-flk6p" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "kube-api-access-flk6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:24.407526 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.407495 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-envoy" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:24.408397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.408375 5946 generic.go:356] "Generic (PLEG): container finished" podID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerID="d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9" exitCode=0 Nov 03 16:39:24.408512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.408486 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-podinfo" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:24.408637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.408580 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" Nov 03 16:39:24.408637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.408602 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" event={"ID":"3176a5ac-4339-4bde-b9e4-db18492adf2e","Type":"ContainerDied","Data":"d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9"} Nov 03 16:39:24.408807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.408648 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" event={"ID":"3176a5ac-4339-4bde-b9e4-db18492adf2e","Type":"ContainerDied","Data":"44fb84e5cdff2191a12bb71cef0c209f69e475151aa08b4d21e499f563c48f91"} Nov 03 16:39:24.409067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.409040 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-token" pod "3176a5ac-4339-4bde-b9e4-db18492adf2e" (UID: "3176a5ac-4339-4bde-b9e4-db18492adf2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:24.411515 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.411493 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" event={"ID":"e3431a37-3f66-47e5-bcff-61ba7ebeeb90","Type":"ContainerDied","Data":"abb15647aa663b22c84586f2a258e5418829b4fb46c52633e3a5285555526cda"} Nov 03 16:39:24.411515 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.411510 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82" Nov 03 16:39:24.460132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.456561 5946 scope.go:122] "RemoveContainer" containerID="ec2d4d85b8cf86717213016776b05f9f65c1c63225e2cbdf4bac0d60d6151b3e" Nov 03 16:39:24.472937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.472913 5946 scope.go:122] "RemoveContainer" containerID="d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9" Nov 03 16:39:24.500606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.500578 5946 scope.go:122] "RemoveContainer" containerID="d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9" Nov 03 16:39:24.500985 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:24.500961 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9\": container with ID starting with d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9 not found: ID does not exist" containerID="d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9" Nov 03 16:39:24.501082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.500993 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9"} err="failed to get container status \"d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9\": rpc error: code = NotFound desc = could not find container \"d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9\": container with ID starting with d606b19616c62c48a8627335927e471ecde79a600eadbfe161027ff93a97c2b9 not found: ID does not exist" Nov 03 16:39:24.501082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.501012 5946 scope.go:122] "RemoveContainer" containerID="9425dcc24f131f0a0aecd8d66d5b0348ee82643ce08e817bfaaaa70df795f301" Nov 03 16:39:24.507771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.507747 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.507771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.507769 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.507968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.507779 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flk6p\" (UniqueName: \"kubernetes.io/projected/3176a5ac-4339-4bde-b9e4-db18492adf2e-kube-api-access-flk6p\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.507968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.507789 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.507968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.507798 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3176a5ac-4339-4bde-b9e4-db18492adf2e-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:24.678563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.678464 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr"] Nov 03 16:39:24.684602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.684571 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-mlflr"] Nov 03 16:39:24.689029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.688713 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.689029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.688766 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 16:39:24.774890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.774857 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:39:24.793801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.793754 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82"] Nov 03 16:39:24.798817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.798788 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-gqs82"] Nov 03 16:39:24.920228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.920150 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c"] Nov 03 16:39:24.927314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.927278 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-d8b6c"] Nov 03 16:39:24.984401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:24.984374 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:39:25.043683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.043644 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr"] Nov 03 16:39:25.054346 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.054311 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr"] Nov 03 16:39:25.055833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.055807 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-credential-socket\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.056200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.056180 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-podinfo\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.056833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.056348 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-credential-socket" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.057225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.057171 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-certs" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.057402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.057383 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-certs\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.057592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.057569 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-kube-api-access-zjp5j\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-kube-api-access-zjp5j\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.057736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.057718 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-envoy\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.058084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.058064 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istiod-ca-cert\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.058288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.058267 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-socket\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.058430 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.058414 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-data\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.058567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.058551 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-token\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-token\") pod \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\" (UID: \"037cce48-42bf-4fe3-9a0a-a5fc6691047f\") " Nov 03 16:39:25.059494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.059473 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.059630 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.059615 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.060116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.060092 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-socket" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.060227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.060111 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-podinfo" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:25.060542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.060507 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-data" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.060883 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.060821 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istiod-ca-cert" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:25.061472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.061444 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-kube-api-access-zjp5j" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "kube-api-access-zjp5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:25.062931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.062906 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-token" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:25.063169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.063139 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-envoy" pod "037cce48-42bf-4fe3-9a0a-a5fc6691047f" (UID: "037cce48-42bf-4fe3-9a0a-a5fc6691047f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.128548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.128524 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:39:25.152160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.152138 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:25.152275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.152184 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-d7mrr" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:25.160589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160565 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-kube-api-access-wnslm\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-kube-api-access-wnslm\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.160715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160622 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-token\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-token\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.160715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160660 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-data\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.160835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160770 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d61d34-53f4-4839-8d58-e36c4506a6c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d61d34-53f4-4839-8d58-e36c4506a6c2-istiod-ca-cert\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.160835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160790 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-credential-socket\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.160953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160909 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-envoy\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.161012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160959 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-certs\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.161012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.160998 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-podinfo\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.161119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161038 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-socket\") pod \"20d61d34-53f4-4839-8d58-e36c4506a6c2\" (UID: \"20d61d34-53f4-4839-8d58-e36c4506a6c2\") " Nov 03 16:39:25.161119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161044 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-credential-socket" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.161119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161063 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-data" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.161309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161131 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d61d34-53f4-4839-8d58-e36c4506a6c2-istiod-ca-cert" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:25.161536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161317 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-certs" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.161594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161468 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-socket" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.161970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161946 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjp5j\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-kube-api-access-zjp5j\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161977 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.161993 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162006 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162020 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162032 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162047 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162060 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d61d34-53f4-4839-8d58-e36c4506a6c2-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162076 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162090 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162106 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037cce48-42bf-4fe3-9a0a-a5fc6691047f-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.162501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.162118 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.163098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.163072 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-envoy" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:25.163249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.163142 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-token" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:25.163249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.163194 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-podinfo" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:25.163343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.163271 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-kube-api-access-wnslm" pod "20d61d34-53f4-4839-8d58-e36c4506a6c2" (UID: "20d61d34-53f4-4839-8d58-e36c4506a6c2"). InnerVolumeSpecName "kube-api-access-wnslm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:25.264125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.264096 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnslm\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-kube-api-access-wnslm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.264125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.264125 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.264368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.264137 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.264368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.264146 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d61d34-53f4-4839-8d58-e36c4506a6c2-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:25.421789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.421756 5946 generic.go:356] "Generic (PLEG): container finished" podID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerID="f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79" exitCode=0 Nov 03 16:39:25.422226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.421810 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" Nov 03 16:39:25.422226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.421844 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" event={"ID":"037cce48-42bf-4fe3-9a0a-a5fc6691047f","Type":"ContainerDied","Data":"f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79"} Nov 03 16:39:25.422226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.421893 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt" event={"ID":"037cce48-42bf-4fe3-9a0a-a5fc6691047f","Type":"ContainerDied","Data":"57a0092051efe6ac6045b46d5d78cb7d5e98812e2aadeb4a728a34cb77621af0"} Nov 03 16:39:25.422226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.421924 5946 scope.go:122] "RemoveContainer" containerID="f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79" Nov 03 16:39:25.424399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.424373 5946 generic.go:356] "Generic (PLEG): container finished" podID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerID="d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53" exitCode=0 Nov 03 16:39:25.424545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.424441 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" event={"ID":"20d61d34-53f4-4839-8d58-e36c4506a6c2","Type":"ContainerDied","Data":"d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53"} Nov 03 16:39:25.424545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.424493 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" event={"ID":"20d61d34-53f4-4839-8d58-e36c4506a6c2","Type":"ContainerDied","Data":"71bcf38f6fe68c040d960ae0e07239e34b146b0a78502429d717980b88a67c0a"} Nov 03 16:39:25.424545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.424502 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9" Nov 03 16:39:25.445148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.445125 5946 scope.go:122] "RemoveContainer" containerID="f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79" Nov 03 16:39:25.445517 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:25.445497 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79\": container with ID starting with f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79 not found: ID does not exist" containerID="f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79" Nov 03 16:39:25.445591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.445528 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79"} err="failed to get container status \"f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79\": rpc error: code = NotFound desc = could not find container \"f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79\": container with ID starting with f55160497defdb335ece35a7f33d325c4de6079b70396b2a3296ba7bd3bc2b79 not found: ID does not exist" Nov 03 16:39:25.445591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.445548 5946 scope.go:122] "RemoveContainer" containerID="d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53" Nov 03 16:39:25.455729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.455708 5946 scope.go:122] "RemoveContainer" containerID="d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53" Nov 03 16:39:25.456009 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:25.455986 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53\": container with ID starting with d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53 not found: ID does not exist" containerID="d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53" Nov 03 16:39:25.456112 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.456014 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53"} err="failed to get container status \"d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53\": rpc error: code = NotFound desc = could not find container \"d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53\": container with ID starting with d81156c0b37c5d383e4ab1b1a02e54fba4c218e347b6126b2d57569aed612a53 not found: ID does not exist" Nov 03 16:39:25.597510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.597418 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9"] Nov 03 16:39:25.607380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.607341 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-hhlx9"] Nov 03 16:39:25.725151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.725112 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt"] Nov 03 16:39:25.733705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.733669 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-wcdrt"] Nov 03 16:39:25.921257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:25.921164 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:26.007551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:26.007518 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" path="/var/lib/kubelet/pods/037cce48-42bf-4fe3-9a0a-a5fc6691047f/volumes" Nov 03 16:39:26.008027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:26.008011 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" path="/var/lib/kubelet/pods/20d61d34-53f4-4839-8d58-e36c4506a6c2/volumes" Nov 03 16:39:26.008491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:26.008470 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" path="/var/lib/kubelet/pods/3176a5ac-4339-4bde-b9e4-db18492adf2e/volumes" Nov 03 16:39:26.009420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:26.009401 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" path="/var/lib/kubelet/pods/7b04a7e3-f1ae-4b8a-863d-c821f4eeb430/volumes" Nov 03 16:39:26.009855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:26.009839 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" path="/var/lib/kubelet/pods/e3431a37-3f66-47e5-bcff-61ba7ebeeb90/volumes" Nov 03 16:39:26.010300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:26.010284 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" path="/var/lib/kubelet/pods/e589bc27-2ffb-4627-8dc4-2ed8a606e716/volumes" Nov 03 16:39:28.211355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.211302 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.237350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.237298 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.424360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.424308 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Nov 03 16:39:28.510234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.509889 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:39:28.510448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.510329 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 16:39:28.514230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.510616 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:39:28.524062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.523315 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.524062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.523589 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 16:39:28.579816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.579766 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.608240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.585306 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.716231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.715290 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.987313 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.983276 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.988426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:28.988375 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ns7gv" podUID="b7bf8082-ce3f-4b37-9e27-b6ae27b4ba63" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:29.586377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:29.586331 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 16:39:29.600297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:29.600265 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:39:29.716736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:29.716703 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:39:30.798622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:30.798594 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:32.696182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:32.696144 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:32.901080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:32.901043 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:33.094907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:33.094872 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:33.661864 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:33.661833 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:33.937075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:33.937004 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:34.222833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:34.222642 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:34.445477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:34.445435 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq"] Nov 03 16:39:34.493579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:34.493547 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b70a8a53-chrlq"] Nov 03 16:39:35.103614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:35.103554 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:36.011894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:36.011856 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47f2889-9b30-4ca2-95e8-fb4e2e1f0764" path="/var/lib/kubelet/pods/b47f2889-9b30-4ca2-95e8-fb4e2e1f0764/volumes" Nov 03 16:39:36.416288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:36.416180 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:37.247189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:37.247114 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:37.500021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:37.499945 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:38.174974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.174907 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:38.662201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.662169 5946 generic.go:356] "Generic (PLEG): container finished" podID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerID="bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a" exitCode=0 Nov 03 16:39:38.662733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.662264 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" event={"ID":"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f","Type":"ContainerDied","Data":"bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a"} Nov 03 16:39:38.662792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.662747 5946 scope.go:122] "RemoveContainer" containerID="bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a" Nov 03 16:39:38.664386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.664367 5946 generic.go:356] "Generic (PLEG): container finished" podID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerID="e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c" exitCode=0 Nov 03 16:39:38.664481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.664444 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" event={"ID":"267d739d-4e26-432a-bd7f-865b6ccbc8df","Type":"ContainerDied","Data":"e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c"} Nov 03 16:39:38.664537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.664511 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:39:38.664631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.664615 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:39:38.664686 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.664670 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:39:38.664888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.664873 5946 scope.go:122] "RemoveContainer" containerID="e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c" Nov 03 16:39:38.672239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.667316 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:39:38.672239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.667474 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:39:38.672239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:38.667577 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:39:39.701337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:39.701308 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" event={"ID":"267d739d-4e26-432a-bd7f-865b6ccbc8df","Type":"ContainerStarted","Data":"be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0"} Nov 03 16:39:39.704525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:39.704493 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" event={"ID":"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f","Type":"ContainerStarted","Data":"920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455"} Nov 03 16:39:39.890638 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:39.890561 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:39:39.891445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:39.891188 5946 patch_prober.go:29] interesting pod/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 16:39:39.891445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:39.891253 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 16:39:40.337234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:40.335586 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:39:40.357233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:40.345334 5946 patch_prober.go:29] interesting pod/e2e-mt-9635db0d-openshift-default-5c876b8797-km687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 16:39:40.357233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:40.345415 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 16:39:40.890721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:40.890688 5946 patch_prober.go:29] interesting pod/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 16:39:40.891134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:40.890739 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 16:39:41.341239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:41.340590 5946 patch_prober.go:29] interesting pod/e2e-mt-9635db0d-openshift-default-5c876b8797-km687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 16:39:41.341239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:41.340653 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 16:39:41.891370 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:41.891328 5946 patch_prober.go:29] interesting pod/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 16:39:41.891884 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:41.891393 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 16:39:42.248232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:42.235762 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:42.350231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:42.350100 5946 patch_prober.go:29] interesting pod/e2e-mt-9635db0d-openshift-default-5c876b8797-km687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 16:39:42.350231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:42.350156 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 16:39:42.461233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:42.461028 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:42.667286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:42.663790 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:42.769233 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:42.769161 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:39:42.891548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:42.891512 5946 patch_prober.go:29] interesting pod/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 16:39:42.892066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:42.891596 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 16:39:43.000616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.000586 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:39:43.163233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.162320 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:39:43.337786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.337608 5946 patch_prober.go:29] interesting pod/e2e-mt-9635db0d-openshift-default-5c876b8797-km687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 16:39:43.337786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.337701 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 16:39:43.500232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.498776 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:43.702760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.702684 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:43.891657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.891623 5946 patch_prober.go:29] interesting pod/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 16:39:43.892191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.891686 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 16:39:43.897749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:43.897725 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:44.330231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:44.321559 5946 patch_prober.go:29] interesting pod/e2e-mt-9635db0d-openshift-default-5c876b8797-km687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 16:39:44.330231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:44.321643 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 16:39:44.891294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:44.891257 5946 patch_prober.go:29] interesting pod/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 16:39:44.891479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:44.891365 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 16:39:45.455695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:45.455658 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:39:45.457987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:45.457958 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:39:45.896069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:45.896037 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:39:45.897373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:45.897348 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 16:39:47.990225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:47.990173 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:48.978671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:48.978624 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:49.373618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.373576 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8"] Nov 03 16:39:49.383481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.383439 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" containerID="cri-o://502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295" gracePeriod=30 Nov 03 16:39:49.434451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.432265 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb"] Nov 03 16:39:49.468522 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.468484 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" containerID="cri-o://d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588" gracePeriod=30 Nov 03 16:39:49.541310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.541147 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5"] Nov 03 16:39:49.541886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.541850 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" containerID="cri-o://58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e" gracePeriod=30 Nov 03 16:39:49.549094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.549062 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g"] Nov 03 16:39:49.549423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:49.549397 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" containerID="cri-o://e2d5037e013f215c4104cf379cf27c002579dae31c1847d4a0e30c1f826b07d5" gracePeriod=30 Nov 03 16:39:51.578673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:51.578636 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:53.442755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:53.442717 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:53.995889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:53.995856 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:54.777941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.777915 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:39:54.869739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.869463 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-token\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.869739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.869563 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-socket\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.869739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.869704 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-kube-api-access-sqs2x\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-kube-api-access-sqs2x\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.869739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.869742 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istiod-ca-cert\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.872676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.869774 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-credential-socket\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.872676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.869884 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-data\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.872676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.869926 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-certs\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.872676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.870006 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-envoy\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.872676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.870088 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-podinfo\") pod \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\" (UID: \"3bb495a0-0a65-49d7-bb59-ee9f47188ff1\") " Nov 03 16:39:54.874593 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.874561 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-data" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.874824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.874800 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-socket" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.875026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.874636 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istiod-ca-cert" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:54.875315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.875026 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-certs" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.875315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.875187 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-credential-socket" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.879576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.879553 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-kube-api-access-sqs2x" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "kube-api-access-sqs2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:54.880654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.880581 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-envoy" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.880803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.880700 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-podinfo" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:54.882362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.882338 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-token" pod "3bb495a0-0a65-49d7-bb59-ee9f47188ff1" (UID: "3bb495a0-0a65-49d7-bb59-ee9f47188ff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:54.893766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.893746 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:39:54.909503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.909446 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:39:54.924627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.924588 5946 generic.go:356] "Generic (PLEG): container finished" podID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerID="502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295" exitCode=0 Nov 03 16:39:54.924714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.924651 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" Nov 03 16:39:54.924714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.924649 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" event={"ID":"3bb495a0-0a65-49d7-bb59-ee9f47188ff1","Type":"ContainerDied","Data":"502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295"} Nov 03 16:39:54.924810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.924755 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8" event={"ID":"3bb495a0-0a65-49d7-bb59-ee9f47188ff1","Type":"ContainerDied","Data":"84dd9e37f6201854fc24642b93a16ea6658487408042607309ab0b426b4c5320"} Nov 03 16:39:54.924810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.924787 5946 scope.go:122] "RemoveContainer" containerID="502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295" Nov 03 16:39:54.926354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.926333 5946 generic.go:356] "Generic (PLEG): container finished" podID="d6e16998-b28e-4e06-b395-74853a18ce79" containerID="d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588" exitCode=0 Nov 03 16:39:54.926487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.926381 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" Nov 03 16:39:54.926487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.926415 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" event={"ID":"d6e16998-b28e-4e06-b395-74853a18ce79","Type":"ContainerDied","Data":"d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588"} Nov 03 16:39:54.926487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.926456 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb" event={"ID":"d6e16998-b28e-4e06-b395-74853a18ce79","Type":"ContainerDied","Data":"e9dc2f3a1c7925b24051259258abed6ac257a72fb03b60434a56b574bdba1a44"} Nov 03 16:39:54.927995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.927974 5946 generic.go:356] "Generic (PLEG): container finished" podID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerID="58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e" exitCode=0 Nov 03 16:39:54.928108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.928021 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" Nov 03 16:39:54.928108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.928072 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" event={"ID":"2c429cbc-c372-4d6e-b5f3-69e8374d8292","Type":"ContainerDied","Data":"58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e"} Nov 03 16:39:54.928276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.928118 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5" event={"ID":"2c429cbc-c372-4d6e-b5f3-69e8374d8292","Type":"ContainerDied","Data":"01b2f77bceae326520b6a17340c98df44d1e66966081b57761914f792d19b2d1"} Nov 03 16:39:54.930382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.930360 5946 generic.go:356] "Generic (PLEG): container finished" podID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerID="e2d5037e013f215c4104cf379cf27c002579dae31c1847d4a0e30c1f826b07d5" exitCode=0 Nov 03 16:39:54.930503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.930433 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" event={"ID":"61028362-d71d-4d86-a1b6-7dca6553c61d","Type":"ContainerDied","Data":"e2d5037e013f215c4104cf379cf27c002579dae31c1847d4a0e30c1f826b07d5"} Nov 03 16:39:54.937847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.937832 5946 scope.go:122] "RemoveContainer" containerID="502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295" Nov 03 16:39:54.938173 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:54.938152 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295\": container with ID starting with 502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295 not found: ID does not exist" containerID="502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295" Nov 03 16:39:54.938265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.938182 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295"} err="failed to get container status \"502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295\": rpc error: code = NotFound desc = could not find container \"502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295\": container with ID starting with 502b6a159dfef8c529148de63b65dca1cbea12b7cf62ba2449d48d9d382bb295 not found: ID does not exist" Nov 03 16:39:54.938265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.938242 5946 scope.go:122] "RemoveContainer" containerID="d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588" Nov 03 16:39:54.942584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.942562 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:39:54.949076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.949061 5946 scope.go:122] "RemoveContainer" containerID="d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588" Nov 03 16:39:54.949334 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:54.949314 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588\": container with ID starting with d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588 not found: ID does not exist" containerID="d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588" Nov 03 16:39:54.949408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.949340 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588"} err="failed to get container status \"d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588\": rpc error: code = NotFound desc = could not find container \"d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588\": container with ID starting with d6cd59bba3c9e24984cb6c189534c119aa113c290faebc3559dd0da78fbea588 not found: ID does not exist" Nov 03 16:39:54.949408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.949355 5946 scope.go:122] "RemoveContainer" containerID="58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e" Nov 03 16:39:54.958909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.958890 5946 scope.go:122] "RemoveContainer" containerID="58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e" Nov 03 16:39:54.959178 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:39:54.959161 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e\": container with ID starting with 58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e not found: ID does not exist" containerID="58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e" Nov 03 16:39:54.959279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.959185 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e"} err="failed to get container status \"58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e\": rpc error: code = NotFound desc = could not find container \"58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e\": container with ID starting with 58867b0f2f2ed69167810732285dd0e85998bced3ab9496eefed665b5417e84e not found: ID does not exist" Nov 03 16:39:54.959279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.959199 5946 scope.go:122] "RemoveContainer" containerID="2be03d1904f66e99730ea4b1566b3eebe8c4c2e6456636a5415119ca7aec7b9e" Nov 03 16:39:54.971980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.971958 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-kube-api-access-g5dzb\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-kube-api-access-g5dzb\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.972110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972001 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-certs\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.972110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972042 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6e16998-b28e-4e06-b395-74853a18ce79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e16998-b28e-4e06-b395-74853a18ce79-istio-podinfo\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.972110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972080 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-data\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.972316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972118 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e16998-b28e-4e06-b395-74853a18ce79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e16998-b28e-4e06-b395-74853a18ce79-istiod-ca-cert\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.972316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972149 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-credential-socket\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.972439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972372 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-certs" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.972439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972398 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-data" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.972439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972427 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-socket\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.972598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972456 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-credential-socket" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.972598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972477 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-token\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-token\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.972598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972515 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-kube-api-access-6z8vp\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-kube-api-access-6z8vp\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.972598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972581 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-certs\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.972825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972615 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-podinfo\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.972825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972673 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-socket" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.972825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972682 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e16998-b28e-4e06-b395-74853a18ce79-istiod-ca-cert" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:54.973106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972866 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-socket\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.973106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972925 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-certs" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.973106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.972932 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istiod-ca-cert\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.973106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973037 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-envoy\") pod \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\" (UID: \"2c429cbc-c372-4d6e-b5f3-69e8374d8292\") " Nov 03 16:39:54.973394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973143 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-credential-socket\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.973394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973316 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-envoy\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.973394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973335 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-socket" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.973567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973407 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-data\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.973567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973434 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-istio-token\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-istio-token\") pod \"d6e16998-b28e-4e06-b395-74853a18ce79\" (UID: \"d6e16998-b28e-4e06-b395-74853a18ce79\") " Nov 03 16:39:54.973567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973558 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-credential-socket" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.973797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973772 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istiod-ca-cert" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:54.973933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.973908 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-data" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.974227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974173 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974230 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974245 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974365 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e16998-b28e-4e06-b395-74853a18ce79-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974379 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974393 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974406 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974422 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974436 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974449 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974462 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974476 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974491 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974506 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974520 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974534 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974547 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974560 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqs2x\" (UniqueName: \"kubernetes.io/projected/3bb495a0-0a65-49d7-bb59-ee9f47188ff1-kube-api-access-sqs2x\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.974942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.974574 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:54.985343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.983703 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-podinfo" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:54.985343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.983703 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-kube-api-access-g5dzb" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "kube-api-access-g5dzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:54.985343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.983754 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-istio-token" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:54.985343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.983790 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-kube-api-access-6z8vp" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "kube-api-access-6z8vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:54.985343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.983998 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-token" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:54.985343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.984170 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-envoy" pod "2c429cbc-c372-4d6e-b5f3-69e8374d8292" (UID: "2c429cbc-c372-4d6e-b5f3-69e8374d8292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.985343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.984330 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-envoy" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:54.987184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:54.987151 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6e16998-b28e-4e06-b395-74853a18ce79-istio-podinfo" pod "d6e16998-b28e-4e06-b395-74853a18ce79" (UID: "d6e16998-b28e-4e06-b395-74853a18ce79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:55.075794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.075762 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-kube-api-access-drpv4\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-kube-api-access-drpv4\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.075835 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-credential-socket\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.075862 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-envoy\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.075905 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-data\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.075989 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-socket\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076123 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-token\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-token\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076161 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-certs\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076272 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-credential-socket" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:55.076300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076292 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-data" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:55.076516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076366 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-socket" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:55.076516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076506 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-certs" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:55.076860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076838 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-podinfo\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.076951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.076910 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61028362-d71d-4d86-a1b6-7dca6553c61d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61028362-d71d-4d86-a1b6-7dca6553c61d-istiod-ca-cert\") pod \"61028362-d71d-4d86-a1b6-7dca6553c61d\" (UID: \"61028362-d71d-4d86-a1b6-7dca6553c61d\") " Nov 03 16:39:55.077759 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077402 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61028362-d71d-4d86-a1b6-7dca6553c61d-istiod-ca-cert" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:55.077867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077767 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e16998-b28e-4e06-b395-74853a18ce79-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.077867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077789 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.077867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077802 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61028362-d71d-4d86-a1b6-7dca6553c61d-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.077867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077817 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5dzb\" (UniqueName: \"kubernetes.io/projected/d6e16998-b28e-4e06-b395-74853a18ce79-kube-api-access-g5dzb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.077867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077831 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e16998-b28e-4e06-b395-74853a18ce79-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.077867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077844 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.077867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077860 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8vp\" (UniqueName: \"kubernetes.io/projected/2c429cbc-c372-4d6e-b5f3-69e8374d8292-kube-api-access-6z8vp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.078132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077874 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.078132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077887 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.078132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077902 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.078132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077915 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.078132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077931 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c429cbc-c372-4d6e-b5f3-69e8374d8292-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.078132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.077946 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.078513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.078488 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-envoy" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:39:55.078623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.078564 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-kube-api-access-drpv4" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "kube-api-access-drpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:55.078939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.078921 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-token" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:55.079353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.079333 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-podinfo" pod "61028362-d71d-4d86-a1b6-7dca6553c61d" (UID: "61028362-d71d-4d86-a1b6-7dca6553c61d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:39:55.179818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.179740 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.179818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.179768 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.179818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.179780 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61028362-d71d-4d86-a1b6-7dca6553c61d-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.179818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.179790 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drpv4\" (UniqueName: \"kubernetes.io/projected/61028362-d71d-4d86-a1b6-7dca6553c61d-kube-api-access-drpv4\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:55.341072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.341034 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8"] Nov 03 16:39:55.346519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.346492 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-sh2x8"] Nov 03 16:39:55.609888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.609856 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb"] Nov 03 16:39:55.610057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.609905 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-tlbvb"] Nov 03 16:39:55.727808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.727765 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5"] Nov 03 16:39:55.742916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.742881 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-fcqv5"] Nov 03 16:39:55.944352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.944257 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" event={"ID":"61028362-d71d-4d86-a1b6-7dca6553c61d","Type":"ContainerDied","Data":"f024f7daa75ae85b7dfd19a108af8b71be591492f539e5e4d26a0177d9c24333"} Nov 03 16:39:55.944352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.944333 5946 scope.go:122] "RemoveContainer" containerID="e2d5037e013f215c4104cf379cf27c002579dae31c1847d4a0e30c1f826b07d5" Nov 03 16:39:55.944811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:55.944269 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g" Nov 03 16:39:56.008347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:56.008307 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" path="/var/lib/kubelet/pods/2c429cbc-c372-4d6e-b5f3-69e8374d8292/volumes" Nov 03 16:39:56.008914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:56.008891 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" path="/var/lib/kubelet/pods/3bb495a0-0a65-49d7-bb59-ee9f47188ff1/volumes" Nov 03 16:39:56.009516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:56.009495 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" path="/var/lib/kubelet/pods/d6e16998-b28e-4e06-b395-74853a18ce79/volumes" Nov 03 16:39:56.193680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:56.193643 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g"] Nov 03 16:39:56.199746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:56.199672 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-8sq8g"] Nov 03 16:39:58.005468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:58.005434 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" path="/var/lib/kubelet/pods/61028362-d71d-4d86-a1b6-7dca6553c61d/volumes" Nov 03 16:39:58.847920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:39:58.847719 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:03.610335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:03.610298 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:08.478815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:08.478776 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:10.281556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.281521 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2"] Nov 03 16:40:10.282082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282068 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282083 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282098 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282103 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282111 5946 cpu_manager.go:414] "containerMap: removing container" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282116 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282123 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282127 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282139 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" Nov 03 16:40:10.282142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282144 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282151 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282156 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282162 5946 cpu_manager.go:414] "containerMap: removing container" podUID="64ad51b6-004e-458b-8851-764b3aae7795" containerName="revoke-keys" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282166 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ad51b6-004e-458b-8851-764b3aae7795" containerName="revoke-keys" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282181 5946 cpu_manager.go:414] "containerMap: removing container" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282185 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282194 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282198 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282228 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282232 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282248 5946 cpu_manager.go:414] "containerMap: removing container" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282252 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282261 5946 cpu_manager.go:414] "containerMap: removing container" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282266 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282360 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d61d34-53f4-4839-8d58-e36c4506a6c2" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282371 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb495a0-0a65-49d7-bb59-ee9f47188ff1" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282379 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3431a37-3f66-47e5-bcff-61ba7ebeeb90" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282388 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282395 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="61028362-d71d-4d86-a1b6-7dca6553c61d" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282402 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c429cbc-c372-4d6e-b5f3-69e8374d8292" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282408 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3176a5ac-4339-4bde-b9e4-db18492adf2e" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282413 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="037cce48-42bf-4fe3-9a0a-a5fc6691047f" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282420 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b04a7e3-f1ae-4b8a-863d-c821f4eeb430" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282427 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e16998-b28e-4e06-b395-74853a18ce79" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282432 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e589bc27-2ffb-4627-8dc4-2ed8a606e716" containerName="istio-proxy" Nov 03 16:40:10.282464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.282438 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ad51b6-004e-458b-8851-764b3aae7795" containerName="revoke-keys" Nov 03 16:40:10.287847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.287825 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.346322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.346295 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2"] Nov 03 16:40:10.393410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.393378 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:40:10.393410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.393378 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:40:10.393410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.393407 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:40:10.393676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.393425 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:40:10.399279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.399242 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.399439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.399417 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.399490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.399460 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c896l\" (UniqueName: \"kubernetes.io/projected/a9e550cc-f09e-4c22-87db-dda7f442ebc6-kube-api-access-c896l\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.500837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.500785 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.501062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.500922 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.501062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.500955 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c896l\" (UniqueName: \"kubernetes.io/projected/a9e550cc-f09e-4c22-87db-dda7f442ebc6-kube-api-access-c896l\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.501574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.501547 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.503561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.503534 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.567712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.567641 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c896l\" (UniqueName: \"kubernetes.io/projected/a9e550cc-f09e-4c22-87db-dda7f442ebc6-kube-api-access-c896l\") pod \"maas-api-revoke-sub-4f39b18f-65lc2\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.601654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.601612 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:10.988800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:10.988771 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2"] Nov 03 16:40:10.989893 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:40:10.989868 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e550cc_f09e_4c22_87db_dda7f442ebc6.slice/crio-df1874a37518cc716f91020421dccddc916f7b22dc51cbd4a9f9f372cf08d004 WatchSource:0}: Error finding container df1874a37518cc716f91020421dccddc916f7b22dc51cbd4a9f9f372cf08d004: Status 404 returned error can't find the container with id df1874a37518cc716f91020421dccddc916f7b22dc51cbd4a9f9f372cf08d004 Nov 03 16:40:11.060463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.060425 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" event={"ID":"a9e550cc-f09e-4c22-87db-dda7f442ebc6","Type":"ContainerStarted","Data":"df1874a37518cc716f91020421dccddc916f7b22dc51cbd4a9f9f372cf08d004"} Nov 03 16:40:11.125657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.125622 5946 scope.go:122] "RemoveContainer" containerID="107d7a0052f2ea8be76ddada2b530ee8cd61277a89afb32855a0046b88290667" Nov 03 16:40:11.136520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.136493 5946 scope.go:122] "RemoveContainer" containerID="c05fd76850c91d087a393d60f543bb9c889c9a9ef9466352e308d2a214e9348d" Nov 03 16:40:11.146720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.146695 5946 scope.go:122] "RemoveContainer" containerID="4b7dc6b78152440cea89bb123cf93b0901740d624a10dd989a4237d48513767c" Nov 03 16:40:11.156563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.156537 5946 scope.go:122] "RemoveContainer" containerID="db6eac3d54eb12e277f65ae836f99dc7038d6363c2e8ec2f00d25bd75a271b66" Nov 03 16:40:11.169078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.169051 5946 scope.go:122] "RemoveContainer" containerID="6b055aeca602d55cdf0fc4b9776abdf720dec83a7cfd2887918ce45cb0c593c2" Nov 03 16:40:11.179194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.179172 5946 scope.go:122] "RemoveContainer" containerID="8da43f51ed31dbc7c203cc6ad709231a6b1c7df9c544d4897762b67a791ee181" Nov 03 16:40:11.188716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.188692 5946 scope.go:122] "RemoveContainer" containerID="8c2ac25ed462365671c0b02d4d2ff6391e297177f1e112df82473c3ac71dc358" Nov 03 16:40:11.199005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:11.198984 5946 scope.go:122] "RemoveContainer" containerID="aca505b3b99d3fd0e3c83d7c7dcfd48b001c8bda2dafa860ca048e801279dccf" Nov 03 16:40:12.069792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:12.069757 5946 generic.go:356] "Generic (PLEG): container finished" podID="a9e550cc-f09e-4c22-87db-dda7f442ebc6" containerID="68d328db32e31ff19ebc991b5b8eb3b897e6d797c4a3bc9ef347adc19fb9305c" exitCode=0 Nov 03 16:40:12.070182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:12.069833 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" event={"ID":"a9e550cc-f09e-4c22-87db-dda7f442ebc6","Type":"ContainerDied","Data":"68d328db32e31ff19ebc991b5b8eb3b897e6d797c4a3bc9ef347adc19fb9305c"} Nov 03 16:40:12.586673 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:40:12.586638 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:40:13.088432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.088395 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:13.257159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.257133 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:13.341281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.341172 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-openshift-ca\") pod \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " Nov 03 16:40:13.341435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.341278 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e550cc-f09e-4c22-87db-dda7f442ebc6-kube-api-access-c896l\" (UniqueName: \"kubernetes.io/projected/a9e550cc-f09e-4c22-87db-dda7f442ebc6-kube-api-access-c896l\") pod \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " Nov 03 16:40:13.341531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.341516 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-xks-ca\") pod \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\" (UID: \"a9e550cc-f09e-4c22-87db-dda7f442ebc6\") " Nov 03 16:40:13.341593 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.341565 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-openshift-ca" pod "a9e550cc-f09e-4c22-87db-dda7f442ebc6" (UID: "a9e550cc-f09e-4c22-87db-dda7f442ebc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:13.342028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.342002 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:40:13.343527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.343509 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e550cc-f09e-4c22-87db-dda7f442ebc6-kube-api-access-c896l" pod "a9e550cc-f09e-4c22-87db-dda7f442ebc6" (UID: "a9e550cc-f09e-4c22-87db-dda7f442ebc6"). InnerVolumeSpecName "kube-api-access-c896l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:13.343752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.343726 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-xks-ca" pod "a9e550cc-f09e-4c22-87db-dda7f442ebc6" (UID: "a9e550cc-f09e-4c22-87db-dda7f442ebc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:13.443369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.443326 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e550cc-f09e-4c22-87db-dda7f442ebc6-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:40:13.443369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.443363 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c896l\" (UniqueName: \"kubernetes.io/projected/a9e550cc-f09e-4c22-87db-dda7f442ebc6-kube-api-access-c896l\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:40:13.670280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:13.670141 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:14.088670 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:14.088641 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" Nov 03 16:40:14.089089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:14.088678 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2" event={"ID":"a9e550cc-f09e-4c22-87db-dda7f442ebc6","Type":"ContainerDied","Data":"df1874a37518cc716f91020421dccddc916f7b22dc51cbd4a9f9f372cf08d004"} Nov 03 16:40:14.089089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:14.088738 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1874a37518cc716f91020421dccddc916f7b22dc51cbd4a9f9f372cf08d004" Nov 03 16:40:14.351420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:14.351330 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:15.051327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:15.050952 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:15.461958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:15.461816 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:18.776520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:18.776486 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:20.024921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:20.024887 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:20.247703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:20.247663 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:21.089637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:21.089607 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:23.386521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:23.386481 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:23.598108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:23.598074 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:24.153608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:24.153564 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:28.954534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:28.954479 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:29.289607 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:29.289554 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:29.714174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:29.714085 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:40:33.975304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:33.975267 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:34.258364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:34.258328 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:38.961666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:38.961625 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:39.160362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:39.160325 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:39.760036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:39.759998 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:40.189110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:40.189032 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:42.586625 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:40:42.586585 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:40:43.993945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:43.993900 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:44.337436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:44.337398 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:44.610794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:44.610718 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:44.816668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:44.816637 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:45.019826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:45.019790 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:45.323666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:45.323583 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:45.512128 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:45.512088 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:46.085978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:46.084607 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:46.512433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:46.512406 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:49.210758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:49.210690 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:49.405350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:49.405311 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:49.604058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:49.604024 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:49.845578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:49.845544 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:50.036795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:50.036764 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:52.033355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:52.033315 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:52.247705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:52.247670 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:54.667465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:54.667346 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:57.434872 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:57.434842 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:57.632266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:57.632228 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:59.809343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:40:59.809184 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:03.913814 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:03.913747 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:04.366925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:04.366885 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:05.012680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:05.012629 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:05.501236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:05.500312 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:05.861273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:05.861154 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:10.130474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:10.130440 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:10.317466 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:10.317437 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:12.620667 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:41:12.620622 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:41:13.499634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:13.499597 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:13.693267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:13.693236 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:15.658444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:15.658381 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:20.555763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:20.555691 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:25.295486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:25.295431 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:29.934894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:29.934816 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:30.146874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:30.146816 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:30.349900 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:30.349822 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:34.327350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:34.327286 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:34.899063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:34.898305 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:39.698004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:39.697135 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:42.586743 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:41:42.586702 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:41:44.489194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:44.489161 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:49.252492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:49.252451 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:51.225745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:51.225711 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:51.436141 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:51.433108 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:51.523354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:51.523317 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:41:53.979531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:53.978295 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:57.063720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:57.063680 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k"] Nov 03 16:41:57.114533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:57.114501 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f29cd44-d6v4k"] Nov 03 16:41:58.005717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:58.005676 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c55f16-1164-40c8-a946-8ed2960e4624" path="/var/lib/kubelet/pods/a9c55f16-1164-40c8-a946-8ed2960e4624/volumes" Nov 03 16:41:58.642672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:58.642637 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:58.925386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:58.925201 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:59.115496 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:41:59.115444 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:02.111372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:02.111326 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:02.397956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:02.397878 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:03.521155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:03.521102 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:08.114756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:08.114729 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:08.328696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:08.328649 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:08.513167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:08.513135 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:10.300641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:10.300604 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:42:10.304143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:10.304117 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:42:11.414890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:11.414855 5946 scope.go:122] "RemoveContainer" containerID="56d124e6382ca58f447d49b1d9e22ba16a3d16516da471a2bfe882e534d32fd4" Nov 03 16:42:11.434611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:11.434580 5946 scope.go:122] "RemoveContainer" containerID="dd3ee77e79232fc906354bd4aebd21f17b56f8e5ce44a068937ee5b18387aa50" Nov 03 16:42:11.444699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:11.444674 5946 scope.go:122] "RemoveContainer" containerID="728dd2b5a6be80e249bb0eda2e62709bcb1f65a3ff6236caaeb22340bae7b41a" Nov 03 16:42:12.587878 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:42:12.587842 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:42:13.362078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:13.362039 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:17.062993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:17.062956 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn"] Nov 03 16:42:17.113563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:17.113527 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f5a6d8-sxrhn"] Nov 03 16:42:18.004704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:18.004668 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ec9cf2-6a82-42ee-9a65-8b467d52fad1" path="/var/lib/kubelet/pods/90ec9cf2-6a82-42ee-9a65-8b467d52fad1/volumes" Nov 03 16:42:18.194855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:18.191943 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:20.821372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:20.821301 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:21.589710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:21.589677 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:22.312741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.312699 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc"] Nov 03 16:42:22.313658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.313626 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a9e550cc-f09e-4c22-87db-dda7f442ebc6" containerName="revoke-keys" Nov 03 16:42:22.313658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.313648 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e550cc-f09e-4c22-87db-dda7f442ebc6" containerName="revoke-keys" Nov 03 16:42:22.313907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.313795 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e550cc-f09e-4c22-87db-dda7f442ebc6" containerName="revoke-keys" Nov 03 16:42:22.318998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.318976 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.375941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.375910 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0fda73-openshift-default-dockercfg-5jmm5\"" Nov 03 16:42:22.430433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.430403 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc"] Nov 03 16:42:22.450649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.450620 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:22.466466 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.466430 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-envoy\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.466753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.466727 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-data\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.466870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.466801 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-certs\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.466870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.466861 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgvg\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-kube-api-access-qtgvg\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.467715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.467451 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-podinfo\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.467715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.467501 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-socket\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.467715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.467531 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-token\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.467715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.467593 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istiod-ca-cert\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.468011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.467827 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-credential-socket\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571108 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-podinfo\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571179 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-socket\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571229 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-token\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571292 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istiod-ca-cert\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571371 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-credential-socket\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571463 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-envoy\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571496 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-data\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571599 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-certs\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.571768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.571659 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgvg\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-kube-api-access-qtgvg\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.572845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.572814 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-credential-socket\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.573069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.573050 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-socket\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.573905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.573878 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istiod-ca-cert\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.574126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.574110 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-data\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.578193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.576633 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-certs\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.585258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.582760 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-podinfo\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.585258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.582971 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-envoy\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.642238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.641998 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:22.647763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.645318 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgvg\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-kube-api-access-qtgvg\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.647763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.645854 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-token\") pod \"e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:22.945318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:22.945201 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:23.374433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:23.374390 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:23.391665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:23.391632 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc"] Nov 03 16:42:23.397087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:23.397041 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:42:23.397246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:23.397224 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:42:23.397319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:23.397281 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:42:23.854696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:23.854669 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:24.333215 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:24.333180 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" event={"ID":"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a","Type":"ContainerStarted","Data":"6354178bb1b7a787fc607209340afaa73ab32dc200e05d774d32ec5d30fe628f"} Nov 03 16:42:24.333413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:24.333264 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" event={"ID":"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a","Type":"ContainerStarted","Data":"a3639497b9a09231dc32cd8202ca7dc744271c5bd5d8f84bc054b3bacaea6657"} Nov 03 16:42:24.510136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:24.510068 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" podStartSLOduration=2.5100509239999997 podStartE2EDuration="2.510050924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:24.45642375 +0000 UTC m=+1814.577476607" watchObservedRunningTime="2026-11-03 16:42:24.510050924 +0000 UTC m=+1814.631103752" Nov 03 16:42:24.946031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:24.945995 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:24.951945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:24.951916 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:25.340894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:25.340856 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:25.341995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:25.341973 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:42:27.344028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:27.343993 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:27.568054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:27.568017 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:28.305448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:28.305410 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:30.620947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:30.620912 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:30.874947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:30.874853 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:31.012672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.012629 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm"] Nov 03 16:42:31.017662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.017632 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.081707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.081677 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm"] Nov 03 16:42:31.100311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.100270 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5nww\" (UniqueName: \"kubernetes.io/projected/c2951083-7d6d-4466-8f76-8f3fcdd047fb-kube-api-access-k5nww\") pod \"payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.100640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.100609 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2951083-7d6d-4466-8f76-8f3fcdd047fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.152294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.151509 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0fda73-dockercfg-tsh2b\"" Nov 03 16:42:31.152294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.151580 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0fda73\"" Nov 03 16:42:31.224235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.213938 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2951083-7d6d-4466-8f76-8f3fcdd047fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.278871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.229098 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2951083-7d6d-4466-8f76-8f3fcdd047fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.282127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.282094 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5nww\" (UniqueName: \"kubernetes.io/projected/c2951083-7d6d-4466-8f76-8f3fcdd047fb-kube-api-access-k5nww\") pod \"payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.319435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.318869 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9"] Nov 03 16:42:31.324466 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.324447 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9"] Nov 03 16:42:31.324816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.324761 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:31.343062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.343032 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5nww\" (UniqueName: \"kubernetes.io/projected/c2951083-7d6d-4466-8f76-8f3fcdd047fb-kube-api-access-k5nww\") pod \"payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.393335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.393304 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvhn\" (UniqueName: \"kubernetes.io/projected/01e504b7-fce1-4fc7-95b1-fc968073f8ad-kube-api-access-7nvhn\") pod \"payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:31.393507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.393483 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e504b7-fce1-4fc7-95b1-fc968073f8ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:31.494819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.494788 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e504b7-fce1-4fc7-95b1-fc968073f8ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:31.494990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.494932 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvhn\" (UniqueName: \"kubernetes.io/projected/01e504b7-fce1-4fc7-95b1-fc968073f8ad-kube-api-access-7nvhn\") pod \"payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:31.495524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.495504 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e504b7-fce1-4fc7-95b1-fc968073f8ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:31.555563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.555534 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvhn\" (UniqueName: \"kubernetes.io/projected/01e504b7-fce1-4fc7-95b1-fc968073f8ad-kube-api-access-7nvhn\") pod \"payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:31.617777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.617741 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:31.636566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.636541 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:31.642644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:31.642620 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:32.026772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.026746 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm"] Nov 03 16:42:32.027775 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:42:32.027748 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2951083_7d6d_4466_8f76_8f3fcdd047fb.slice/crio-47601d6b4fbb121a5f997ef557868ed6d47e9d930a71e5192ad979a2a2882e23 WatchSource:0}: Error finding container 47601d6b4fbb121a5f997ef557868ed6d47e9d930a71e5192ad979a2a2882e23: Status 404 returned error can't find the container with id 47601d6b4fbb121a5f997ef557868ed6d47e9d930a71e5192ad979a2a2882e23 Nov 03 16:42:32.074442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.074416 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9"] Nov 03 16:42:32.087607 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:42:32.087584 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e504b7_fce1_4fc7_95b1_fc968073f8ad.slice/crio-6e3ffcb1bd86646fac05f7a408a50916034bcd1c3f6151ad7a21fc1e0a2ab5d8 WatchSource:0}: Error finding container 6e3ffcb1bd86646fac05f7a408a50916034bcd1c3f6151ad7a21fc1e0a2ab5d8: Status 404 returned error can't find the container with id 6e3ffcb1bd86646fac05f7a408a50916034bcd1c3f6151ad7a21fc1e0a2ab5d8 Nov 03 16:42:32.390725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.390626 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" event={"ID":"c2951083-7d6d-4466-8f76-8f3fcdd047fb","Type":"ContainerStarted","Data":"71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647"} Nov 03 16:42:32.390725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.390685 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" event={"ID":"c2951083-7d6d-4466-8f76-8f3fcdd047fb","Type":"ContainerStarted","Data":"47601d6b4fbb121a5f997ef557868ed6d47e9d930a71e5192ad979a2a2882e23"} Nov 03 16:42:32.390725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.390717 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:32.392274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.392242 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" event={"ID":"01e504b7-fce1-4fc7-95b1-fc968073f8ad","Type":"ContainerStarted","Data":"93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5"} Nov 03 16:42:32.392393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.392298 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" event={"ID":"01e504b7-fce1-4fc7-95b1-fc968073f8ad","Type":"ContainerStarted","Data":"6e3ffcb1bd86646fac05f7a408a50916034bcd1c3f6151ad7a21fc1e0a2ab5d8"} Nov 03 16:42:32.392393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.392338 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:32.679802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.678715 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" podStartSLOduration=2.6786939690000002 podStartE2EDuration="2.678693969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:32.550026046 +0000 UTC m=+1822.671078869" watchObservedRunningTime="2026-11-03 16:42:32.678693969 +0000 UTC m=+1822.799746797" Nov 03 16:42:32.839721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.839658 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" podStartSLOduration=1.8396375809999999 podStartE2EDuration="1.839637581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:32.67850717 +0000 UTC m=+1822.799559994" watchObservedRunningTime="2026-11-03 16:42:32.839637581 +0000 UTC m=+1822.960690405" Nov 03 16:42:32.842362 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:32.842328 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:42:33.431668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:33.431622 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:33.633473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:33.633434 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:34.027499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:34.027467 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:34.709041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:34.709003 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:38.406385 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:38.406344 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:41.134977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:41.134940 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:41.339085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:41.339054 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:42.586687 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:42:42.586650 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:42:43.281837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:43.281804 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:43.432048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:43.432020 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:42:43.432302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:43.432088 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:42:44.658993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.658915 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7151-160c14"] Nov 03 16:42:44.662852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.662834 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:42:44.721000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.720955 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7151-160c14"] Nov 03 16:42:44.766646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.766612 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8nh5\"" Nov 03 16:42:44.809337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.809307 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc6g\" (UniqueName: \"kubernetes.io/projected/a2d04dad-160a-4dc0-87c5-bf2808b5895b-kube-api-access-9cc6g\") pod \"test-curl-7151-160c14\" (UID: \"a2d04dad-160a-4dc0-87c5-bf2808b5895b\") " pod="opendatahub/test-curl-7151-160c14" Nov 03 16:42:44.910833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.910757 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc6g\" (UniqueName: \"kubernetes.io/projected/a2d04dad-160a-4dc0-87c5-bf2808b5895b-kube-api-access-9cc6g\") pod \"test-curl-7151-160c14\" (UID: \"a2d04dad-160a-4dc0-87c5-bf2808b5895b\") " pod="opendatahub/test-curl-7151-160c14" Nov 03 16:42:44.971172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.971138 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc6g\" (UniqueName: \"kubernetes.io/projected/a2d04dad-160a-4dc0-87c5-bf2808b5895b-kube-api-access-9cc6g\") pod \"test-curl-7151-160c14\" (UID: \"a2d04dad-160a-4dc0-87c5-bf2808b5895b\") " pod="opendatahub/test-curl-7151-160c14" Nov 03 16:42:44.976434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:44.976414 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:42:45.362550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:45.362519 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7151-160c14"] Nov 03 16:42:45.364165 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:42:45.364131 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d04dad_160a_4dc0_87c5_bf2808b5895b.slice/crio-15ece61483d0f3c18769270baf2009fba5c2d7e8550e308c26774cb528d7ce62 WatchSource:0}: Error finding container 15ece61483d0f3c18769270baf2009fba5c2d7e8550e308c26774cb528d7ce62: Status 404 returned error can't find the container with id 15ece61483d0f3c18769270baf2009fba5c2d7e8550e308c26774cb528d7ce62 Nov 03 16:42:45.531267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:45.531232 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7151-160c14" event={"ID":"a2d04dad-160a-4dc0-87c5-bf2808b5895b","Type":"ContainerStarted","Data":"9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85"} Nov 03 16:42:45.531440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:45.531280 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7151-160c14" event={"ID":"a2d04dad-160a-4dc0-87c5-bf2808b5895b","Type":"ContainerStarted","Data":"15ece61483d0f3c18769270baf2009fba5c2d7e8550e308c26774cb528d7ce62"} Nov 03 16:42:45.650386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:45.650273 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7151-160c14" podStartSLOduration=1.6502587850000001 podStartE2EDuration="1.650258785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:45.648823619 +0000 UTC m=+1835.769876442" watchObservedRunningTime="2026-11-03 16:42:45.650258785 +0000 UTC m=+1835.771311607" Nov 03 16:42:48.074177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:48.071919 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:52.814003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:52.811750 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:57.607238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:42:57.605735 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:02.512523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:02.512478 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:03.405438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:03.405392 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:04.120404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.120369 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:04.617546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.617506 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg"] Nov 03 16:43:04.622296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.622239 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.680408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.679591 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-66f145-openshift-default-dockercfg-jn9p5\"" Nov 03 16:43:04.868300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.824652 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-token\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.868811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.841844 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:04.868811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.861066 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg"] Nov 03 16:43:04.897658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.897622 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-data\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.898006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.897976 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-envoy\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.898231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.898190 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-certs\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.898380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.898364 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istiod-ca-cert\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.898504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.898489 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-credential-socket\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.898692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.898678 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-socket\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.899015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.898991 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr4k\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-kube-api-access-prr4k\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:04.906258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:04.899158 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-podinfo\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.005777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.005745 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr4k\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-kube-api-access-prr4k\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.005777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.005790 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-podinfo\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.005842 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-token\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.005954 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-data\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006000 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-envoy\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006036 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-certs\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006059 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istiod-ca-cert\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006083 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-credential-socket\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006160 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-socket\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006492 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-data\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006620 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-socket\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.006892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.006869 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-credential-socket\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.007061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.007012 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istiod-ca-cert\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.007357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.007333 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-certs\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.009103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.009080 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-envoy\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.009233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.009099 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-podinfo\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.071659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.071626 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-token\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.071958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.071935 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr4k\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-kube-api-access-prr4k\") pod \"e2e-api-b-66f145-openshift-default-b8654467-g58fg\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.264684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.264586 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:05.310415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.310379 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:05.724983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.724952 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg"] Nov 03 16:43:05.726785 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:43:05.726751 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ca2e4c_c89f_4b9a_a774_3bf2777bad9e.slice/crio-aefa8650d32cfa988e34da95a7187e7fad6e98860b0e5450958fad9887fe0bda WatchSource:0}: Error finding container aefa8650d32cfa988e34da95a7187e7fad6e98860b0e5450958fad9887fe0bda: Status 404 returned error can't find the container with id aefa8650d32cfa988e34da95a7187e7fad6e98860b0e5450958fad9887fe0bda Nov 03 16:43:05.739871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.730109 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:05.739871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.730288 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:05.739871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.730381 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:05.859412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.859371 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" event={"ID":"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e","Type":"ContainerStarted","Data":"395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299"} Nov 03 16:43:05.859591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.859475 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" event={"ID":"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e","Type":"ContainerStarted","Data":"aefa8650d32cfa988e34da95a7187e7fad6e98860b0e5450958fad9887fe0bda"} Nov 03 16:43:05.986791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:05.986735 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podStartSLOduration=1.9867211299999998 podStartE2EDuration="1.98672113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:05.985525106 +0000 UTC m=+1856.106577927" watchObservedRunningTime="2026-11-03 16:43:05.98672113 +0000 UTC m=+1856.107773953" Nov 03 16:43:06.265708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:06.265620 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:06.266973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:06.266944 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 16:43:06.267113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:06.267017 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 16:43:07.269788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:07.269755 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:07.634444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:07.632785 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:07.874012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:07.873981 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:07.875068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:07.875047 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:43:09.450114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:09.450078 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:09.645511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:09.645464 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:11.502098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:11.502063 5946 scope.go:122] "RemoveContainer" containerID="df5a5b0d8b832b44df97c24cfb714e70a80129dad56b48a5cebafeca9d5fa800" Nov 03 16:43:11.512741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:11.512722 5946 scope.go:122] "RemoveContainer" containerID="c6cb2b4cc722635f0af460e70673f3fb834bc748cbc56f00f5f80d51e2125803" Nov 03 16:43:11.522479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:11.522461 5946 scope.go:122] "RemoveContainer" containerID="1ebf51d1985096423d3aeea51162b20f32c62384be6fabeea6f2547c0b5eb0b5" Nov 03 16:43:12.586324 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:43:12.586289 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:43:12.903158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:12.902989 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:14.070462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.067910 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:14.271482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.271451 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:14.385758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.385656 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl"] Nov 03 16:43:14.393057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.393015 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:14.445329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.445287 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl"] Nov 03 16:43:14.465899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.465678 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1665afc-3f47-47e5-854d-051b26ba4782-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:14.465899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.465778 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w64w\" (UniqueName: \"kubernetes.io/projected/a1665afc-3f47-47e5-854d-051b26ba4782-kube-api-access-7w64w\") pod \"payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:14.504231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.503689 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-66f145\"" Nov 03 16:43:14.519232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.502754 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-66f145-dockercfg-w8zf2\"" Nov 03 16:43:14.599232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.589020 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1665afc-3f47-47e5-854d-051b26ba4782-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:14.599232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.589100 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w64w\" (UniqueName: \"kubernetes.io/projected/a1665afc-3f47-47e5-854d-051b26ba4782-kube-api-access-7w64w\") pod \"payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:14.680980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.676991 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1665afc-3f47-47e5-854d-051b26ba4782-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:14.693352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.692733 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w64w\" (UniqueName: \"kubernetes.io/projected/a1665afc-3f47-47e5-854d-051b26ba4782-kube-api-access-7w64w\") pod \"payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:14.710836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:14.710809 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:15.096606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:15.096578 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl"] Nov 03 16:43:15.096966 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:43:15.096759 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1665afc_3f47_47e5_854d_051b26ba4782.slice/crio-1575922eb8f2de55cb528cf3ebc23dfd013ef012acd4a21234030036756f7bd6 WatchSource:0}: Error finding container 1575922eb8f2de55cb528cf3ebc23dfd013ef012acd4a21234030036756f7bd6: Status 404 returned error can't find the container with id 1575922eb8f2de55cb528cf3ebc23dfd013ef012acd4a21234030036756f7bd6 Nov 03 16:43:15.941669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:15.941633 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" event={"ID":"a1665afc-3f47-47e5-854d-051b26ba4782","Type":"ContainerStarted","Data":"54204414457e1532274c571ab1963ee405971e34d712eb7503f7481c207fd656"} Nov 03 16:43:15.941867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:15.941689 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" event={"ID":"a1665afc-3f47-47e5-854d-051b26ba4782","Type":"ContainerStarted","Data":"1575922eb8f2de55cb528cf3ebc23dfd013ef012acd4a21234030036756f7bd6"} Nov 03 16:43:15.941867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:15.941720 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:16.064505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:16.064454 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" podStartSLOduration=2.064439894 podStartE2EDuration="2.064439894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:16.062647797 +0000 UTC m=+1866.183700622" watchObservedRunningTime="2026-11-03 16:43:16.064439894 +0000 UTC m=+1866.185492717" Nov 03 16:43:17.906294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:17.906253 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:18.097536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:18.097504 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:21.380646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:21.380607 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:43:22.853923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:22.853894 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:26.952609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:26.952571 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:43:27.485428 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:27.485375 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:29.943610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:29.943578 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl"] Nov 03 16:43:29.944382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:29.943783 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" containerID="cri-o://59e041301d52c3192642df3e7ecf64e2e8358326dee9489f58663f41fcec4304" gracePeriod=30 Nov 03 16:43:32.329280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:32.329174 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:33.172532 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:33.172496 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:43:33.172761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:33.172569 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:43:34.065822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:34.065790 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m"] Nov 03 16:43:34.116660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:34.116631 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab49f2ef-ndn9m"] Nov 03 16:43:35.066960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.066927 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:35.081629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.081593 5946 generic.go:356] "Generic (PLEG): container finished" podID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerID="59e041301d52c3192642df3e7ecf64e2e8358326dee9489f58663f41fcec4304" exitCode=0 Nov 03 16:43:35.081782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.081671 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" event={"ID":"7fccc840-a6fe-4506-a7f0-da112c251f2c","Type":"ContainerDied","Data":"59e041301d52c3192642df3e7ecf64e2e8358326dee9489f58663f41fcec4304"} Nov 03 16:43:35.237722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.237695 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:43:35.390075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390035 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-envoy\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390127 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-kube-api-access-tncsl\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-kube-api-access-tncsl\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390162 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fccc840-a6fe-4506-a7f0-da112c251f2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fccc840-a6fe-4506-a7f0-da112c251f2c-istiod-ca-cert\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390201 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-podinfo\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390296 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-socket\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390395 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-token\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-token\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390457 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-data\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390561 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-credential-socket\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390680 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-certs\") pod \"7fccc840-a6fe-4506-a7f0-da112c251f2c\" (UID: \"7fccc840-a6fe-4506-a7f0-da112c251f2c\") " Nov 03 16:43:35.390759 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390684 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fccc840-a6fe-4506-a7f0-da112c251f2c-istiod-ca-cert" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:35.390846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390809 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-data" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:35.390954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.390931 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-credential-socket" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:35.391089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.391064 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-socket" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:35.391182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.391157 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-certs" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:35.391650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.391627 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.391724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.391662 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fccc840-a6fe-4506-a7f0-da112c251f2c-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.391724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.391680 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.391724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.391694 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.391724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.391707 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.392572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.392543 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-kube-api-access-tncsl" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "kube-api-access-tncsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:35.392786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.392766 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-token" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:35.392996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.392972 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-podinfo" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:43:35.393104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.392970 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-envoy" pod "7fccc840-a6fe-4506-a7f0-da112c251f2c" (UID: "7fccc840-a6fe-4506-a7f0-da112c251f2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:35.493363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.493337 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.493363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.493362 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.493363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.493372 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tncsl\" (UniqueName: \"kubernetes.io/projected/7fccc840-a6fe-4506-a7f0-da112c251f2c-kube-api-access-tncsl\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:35.493621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:35.493381 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fccc840-a6fe-4506-a7f0-da112c251f2c-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:36.007429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:36.007397 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2650b82e-ba36-459b-8f2d-3d6b1d5854d8" path="/var/lib/kubelet/pods/2650b82e-ba36-459b-8f2d-3d6b1d5854d8/volumes" Nov 03 16:43:36.090170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:36.090144 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" Nov 03 16:43:36.090580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:36.090181 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl" event={"ID":"7fccc840-a6fe-4506-a7f0-da112c251f2c","Type":"ContainerDied","Data":"f57f05fe77c20a80187e4cd603bd4410ee8fe9249c59e8978d3c15a39a0074ac"} Nov 03 16:43:36.090580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:36.090262 5946 scope.go:122] "RemoveContainer" containerID="59e041301d52c3192642df3e7ecf64e2e8358326dee9489f58663f41fcec4304" Nov 03 16:43:36.321975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:36.321870 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl"] Nov 03 16:43:36.372435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:36.372378 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-wdgxl"] Nov 03 16:43:37.199857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:37.199810 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:37.446052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:37.446010 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:37.878387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:37.878353 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:38.005331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.005301 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" path="/var/lib/kubelet/pods/7fccc840-a6fe-4506-a7f0-da112c251f2c/volumes" Nov 03 16:43:38.563604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.561887 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6"] Nov 03 16:43:38.563604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.562947 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" Nov 03 16:43:38.563604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.562971 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" Nov 03 16:43:38.618230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.615429 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fccc840-a6fe-4506-a7f0-da112c251f2c" containerName="istio-proxy" Nov 03 16:43:38.630445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.630410 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643454 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-envoy\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643500 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-data\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643639 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-certs\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643688 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-socket\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643716 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-podinfo\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643748 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e436b7-0ecc-45f2-9418-d110fadc4e67-istiod-ca-cert\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643807 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhb4\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-kube-api-access-qvhb4\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643936 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-credential-socket\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.643989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.643975 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-token\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.662805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.662771 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6"] Nov 03 16:43:38.706105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.706068 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0ea8a5-openshift-default-dockercfg-zsnwr\"" Nov 03 16:43:38.708722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.708699 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:38.746413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.746370 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-certs\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.753379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.753328 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-socket\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.768419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.768380 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-podinfo\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.768713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.768687 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e436b7-0ecc-45f2-9418-d110fadc4e67-istiod-ca-cert\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.768911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.768893 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhb4\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-kube-api-access-qvhb4\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.769153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.769136 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-credential-socket\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.769331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.769313 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-token\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.769576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.769555 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-envoy\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.769723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.769696 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-credential-socket\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.769807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.769699 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-data\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.769906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.754013 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-socket\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.770001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.769978 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e436b7-0ecc-45f2-9418-d110fadc4e67-istiod-ca-cert\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.770405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.770378 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-data\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.770534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.770424 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-certs\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.772787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.772761 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-podinfo\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.772952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.772929 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-envoy\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.833912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.831572 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-token\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.847937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.847819 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhb4\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-kube-api-access-qvhb4\") pod \"e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:38.910358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.910327 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:38.951307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:38.951279 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:39.396658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:39.396627 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6"] Nov 03 16:43:39.398389 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:43:39.398362 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e436b7_0ecc_45f2_9418_d110fadc4e67.slice/crio-34033e1fadbe89576c07d87d5580e214351d3e755a89e30be7b8389778b36662 WatchSource:0}: Error finding container 34033e1fadbe89576c07d87d5580e214351d3e755a89e30be7b8389778b36662: Status 404 returned error can't find the container with id 34033e1fadbe89576c07d87d5580e214351d3e755a89e30be7b8389778b36662 Nov 03 16:43:39.401394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:39.401359 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:39.401486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:39.401428 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:39.401486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:39.401458 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:40.122188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:40.122151 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" event={"ID":"10e436b7-0ecc-45f2-9418-d110fadc4e67","Type":"ContainerStarted","Data":"df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70"} Nov 03 16:43:40.122640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:40.122229 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" event={"ID":"10e436b7-0ecc-45f2-9418-d110fadc4e67","Type":"ContainerStarted","Data":"34033e1fadbe89576c07d87d5580e214351d3e755a89e30be7b8389778b36662"} Nov 03 16:43:40.304009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:40.303959 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podStartSLOduration=2.303944688 podStartE2EDuration="2.303944688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:40.302090045 +0000 UTC m=+1890.423142878" watchObservedRunningTime="2026-11-03 16:43:40.303944688 +0000 UTC m=+1890.424997507" Nov 03 16:43:40.952438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:40.952402 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:40.958798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:40.958772 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:41.129540 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:41.129512 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:41.130717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:41.130685 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:43:42.187994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:42.187960 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:42.587157 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:43:42.587117 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:43:43.929565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:43.929490 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:44.125797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:44.125649 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:44.323942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:44.323909 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:46.566255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:46.566201 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7151-160c14"] Nov 03 16:43:46.566790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:46.566397 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7151-160c14" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" containerName="test-curl-7151-160c14" containerID="cri-o://9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85" gracePeriod=2 Nov 03 16:43:46.574271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:46.574237 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7151-160c14"] Nov 03 16:43:46.619161 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:43:46.619116 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7151-160c14\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:43:47.109581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.109545 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76"] Nov 03 16:43:47.110535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.110513 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" containerName="test-curl-7151-160c14" Nov 03 16:43:47.110636 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.110538 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" containerName="test-curl-7151-160c14" Nov 03 16:43:47.110766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.110753 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" containerName="test-curl-7151-160c14" Nov 03 16:43:47.114780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.114761 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.176416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.176384 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76"] Nov 03 16:43:47.199270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.199181 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.199449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.199330 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgw6\" (UniqueName: \"kubernetes.io/projected/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-kube-api-access-7vgw6\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.199563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.199538 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.222492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.222468 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:43:47.222632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.222528 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:43:47.222632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.222568 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:43:47.222632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.222571 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:43:47.289360 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:43:47.289326 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7151-160c14\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:43:47.301156 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.301116 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.301366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.301315 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.301417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.301396 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgw6\" (UniqueName: \"kubernetes.io/projected/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-kube-api-access-7vgw6\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.301890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.301868 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.303896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.303879 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.367795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.367713 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgw6\" (UniqueName: \"kubernetes.io/projected/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-kube-api-access-7vgw6\") pod \"maas-api-revoke-keys-e2e-mt-9635db0d-v6r76\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.430358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.430326 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:47.431321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.431300 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:47.835091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:47.835062 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76"] Nov 03 16:43:47.836024 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:43:47.836003 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc3763a_23ed_4817_9d66_ca1bfc3cd192.slice/crio-6289f6c6aa92f517cdd49ab7394b7bf6603e803f297e165a2789adf3a78e44b5 WatchSource:0}: Error finding container 6289f6c6aa92f517cdd49ab7394b7bf6603e803f297e165a2789adf3a78e44b5: Status 404 returned error can't find the container with id 6289f6c6aa92f517cdd49ab7394b7bf6603e803f297e165a2789adf3a78e44b5 Nov 03 16:43:48.182674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:48.182637 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" event={"ID":"8cc3763a-23ed-4817-9d66-ca1bfc3cd192","Type":"ContainerStarted","Data":"5fab07e37ab4cdee8d153182d11c49f4470547a6cef083e7703e7484e84d7da2"} Nov 03 16:43:48.182853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:48.182696 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" event={"ID":"8cc3763a-23ed-4817-9d66-ca1bfc3cd192","Type":"ContainerStarted","Data":"6289f6c6aa92f517cdd49ab7394b7bf6603e803f297e165a2789adf3a78e44b5"} Nov 03 16:43:48.310085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:48.310025 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" podStartSLOduration=1.31000592 podStartE2EDuration="1.31000592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:48.306601342 +0000 UTC m=+1898.427654177" watchObservedRunningTime="2026-11-03 16:43:48.31000592 +0000 UTC m=+1898.431058744" Nov 03 16:43:48.822773 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:48.822704 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:43:48.926644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:48.926614 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d04dad-160a-4dc0-87c5-bf2808b5895b-kube-api-access-9cc6g\" (UniqueName: \"kubernetes.io/projected/a2d04dad-160a-4dc0-87c5-bf2808b5895b-kube-api-access-9cc6g\") pod \"a2d04dad-160a-4dc0-87c5-bf2808b5895b\" (UID: \"a2d04dad-160a-4dc0-87c5-bf2808b5895b\") " Nov 03 16:43:48.928889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:48.928866 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d04dad-160a-4dc0-87c5-bf2808b5895b-kube-api-access-9cc6g" pod "a2d04dad-160a-4dc0-87c5-bf2808b5895b" (UID: "a2d04dad-160a-4dc0-87c5-bf2808b5895b"). InnerVolumeSpecName "kube-api-access-9cc6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:49.028789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.028749 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cc6g\" (UniqueName: \"kubernetes.io/projected/a2d04dad-160a-4dc0-87c5-bf2808b5895b-kube-api-access-9cc6g\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:49.190737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.190648 5946 generic.go:356] "Generic (PLEG): container finished" podID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" containerID="9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85" exitCode=137 Nov 03 16:43:49.190737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.190693 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:43:49.190976 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.190750 5946 scope.go:122] "RemoveContainer" containerID="9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85" Nov 03 16:43:49.192880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.192853 5946 generic.go:356] "Generic (PLEG): container finished" podID="8cc3763a-23ed-4817-9d66-ca1bfc3cd192" containerID="5fab07e37ab4cdee8d153182d11c49f4470547a6cef083e7703e7484e84d7da2" exitCode=0 Nov 03 16:43:49.192960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.192938 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" event={"ID":"8cc3763a-23ed-4817-9d66-ca1bfc3cd192","Type":"ContainerDied","Data":"5fab07e37ab4cdee8d153182d11c49f4470547a6cef083e7703e7484e84d7da2"} Nov 03 16:43:49.201338 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.201320 5946 scope.go:122] "RemoveContainer" containerID="9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85" Nov 03 16:43:49.201652 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:43:49.201633 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85\": container with ID starting with 9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85 not found: ID does not exist" containerID="9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85" Nov 03 16:43:49.201717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:49.201662 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85"} err="failed to get container status \"9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85\": rpc error: code = NotFound desc = could not find container \"9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85\": container with ID starting with 9d86d0884aa79469e7683063c6f8b29be2231216bf0f429abc2d949c0d86ce85 not found: ID does not exist" Nov 03 16:43:49.364452 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:43:49.364416 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7151-160c14\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:43:50.008548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.008495 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" path="/var/lib/kubelet/pods/a2d04dad-160a-4dc0-87c5-bf2808b5895b/volumes" Nov 03 16:43:50.109134 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:43:50.109091 5946 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7151-160c14\" is forbidden: User \"system:node:ip-10-0-135-216.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-216.us-west-2.compute.internal' and this object" podUID="a2d04dad-160a-4dc0-87c5-bf2808b5895b" pod="opendatahub/test-curl-7151-160c14" Nov 03 16:43:50.374633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.374606 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:50.446841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.446810 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-xks-ca\") pod \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " Nov 03 16:43:50.447022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.446861 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-openshift-ca\") pod \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " Nov 03 16:43:50.447085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.447050 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-kube-api-access-7vgw6\" (UniqueName: \"kubernetes.io/projected/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-kube-api-access-7vgw6\") pod \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\" (UID: \"8cc3763a-23ed-4817-9d66-ca1bfc3cd192\") " Nov 03 16:43:50.447252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.447191 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-openshift-ca" pod "8cc3763a-23ed-4817-9d66-ca1bfc3cd192" (UID: "8cc3763a-23ed-4817-9d66-ca1bfc3cd192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:50.447800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.447781 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:50.449073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.449045 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-xks-ca" pod "8cc3763a-23ed-4817-9d66-ca1bfc3cd192" (UID: "8cc3763a-23ed-4817-9d66-ca1bfc3cd192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:43:50.449201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.449186 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-kube-api-access-7vgw6" pod "8cc3763a-23ed-4817-9d66-ca1bfc3cd192" (UID: "8cc3763a-23ed-4817-9d66-ca1bfc3cd192"). InnerVolumeSpecName "kube-api-access-7vgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:50.549780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.549690 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgw6\" (UniqueName: \"kubernetes.io/projected/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-kube-api-access-7vgw6\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:50.549780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:50.549726 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc3763a-23ed-4817-9d66-ca1bfc3cd192-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:43:51.211477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:51.211448 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" Nov 03 16:43:51.211952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:51.211506 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76" event={"ID":"8cc3763a-23ed-4817-9d66-ca1bfc3cd192","Type":"ContainerDied","Data":"6289f6c6aa92f517cdd49ab7394b7bf6603e803f297e165a2789adf3a78e44b5"} Nov 03 16:43:51.211952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:51.211565 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6289f6c6aa92f517cdd49ab7394b7bf6603e803f297e165a2789adf3a78e44b5" Nov 03 16:43:52.475857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:52.475825 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:53.072870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:53.072831 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:53.287041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:53.287009 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:53.558505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:53.558468 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:54.315285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.315250 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:54.783942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.783905 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v"] Nov 03 16:43:54.784645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.784627 5946 cpu_manager.go:414] "containerMap: removing container" podUID="8cc3763a-23ed-4817-9d66-ca1bfc3cd192" containerName="revoke-keys" Nov 03 16:43:54.784719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.784647 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc3763a-23ed-4817-9d66-ca1bfc3cd192" containerName="revoke-keys" Nov 03 16:43:54.784815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.784804 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc3763a-23ed-4817-9d66-ca1bfc3cd192" containerName="revoke-keys" Nov 03 16:43:54.788531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.788511 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.847438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.847397 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6822cb-openshift-default-dockercfg-vpchv\"" Nov 03 16:43:54.951234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.937127 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:54.951638 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.951106 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-socket\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.961340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.961303 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft42s\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-kube-api-access-ft42s\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.961612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.961592 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-certs\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.968240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.961854 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-credential-socket\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.968553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.968529 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-envoy\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.981320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.981286 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-data\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.981555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.981535 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-podinfo\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.981712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.981696 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-token\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:54.981932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:54.981918 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f08a11-6683-4b7b-a297-7a75aa981220-istiod-ca-cert\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.089762 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.089654 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-credential-socket\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.089762 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.089785 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-envoy\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.090068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.089815 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-data\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.090068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.089842 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-podinfo\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.090068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.089891 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-token\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.090314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.090198 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f08a11-6683-4b7b-a297-7a75aa981220-istiod-ca-cert\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.090314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.090267 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-socket\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.090429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.090299 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft42s\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-kube-api-access-ft42s\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.090429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.090420 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-certs\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.091600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.090686 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-data\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.091600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.090743 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-certs\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.091600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.090931 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-credential-socket\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.091867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.091815 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f08a11-6683-4b7b-a297-7a75aa981220-istiod-ca-cert\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.092301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.092277 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-socket\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.100977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.097710 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-envoy\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.102263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.102235 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-podinfo\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.140164 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.139577 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v"] Nov 03 16:43:55.155547 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.153996 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-token\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.155786 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.155757 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft42s\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-kube-api-access-ft42s\") pod \"e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.405614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.405499 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:55.467246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.467151 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:55.888022 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:43:55.887983 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f08a11_6683_4b7b_a297_7a75aa981220.slice/crio-059cac0978ce73a0dd7c77aecf302e1c14aacd909f1ec20b512346036f9b37e7 WatchSource:0}: Error finding container 059cac0978ce73a0dd7c77aecf302e1c14aacd909f1ec20b512346036f9b37e7: Status 404 returned error can't find the container with id 059cac0978ce73a0dd7c77aecf302e1c14aacd909f1ec20b512346036f9b37e7 Nov 03 16:43:55.888022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.887995 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v"] Nov 03 16:43:55.891567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.891527 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:55.891655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.891615 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:55.891703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:55.891659 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:43:56.262942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.253895 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" event={"ID":"d9f08a11-6683-4b7b-a297-7a75aa981220","Type":"ContainerStarted","Data":"e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730"} Nov 03 16:43:56.262942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.253955 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" event={"ID":"d9f08a11-6683-4b7b-a297-7a75aa981220","Type":"ContainerStarted","Data":"059cac0978ce73a0dd7c77aecf302e1c14aacd909f1ec20b512346036f9b37e7"} Nov 03 16:43:56.378555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.378501 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podStartSLOduration=2.378486155 podStartE2EDuration="2.378486155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:56.377029726 +0000 UTC m=+1906.498082550" watchObservedRunningTime="2026-11-03 16:43:56.378486155 +0000 UTC m=+1906.499539052" Nov 03 16:43:56.412806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.412763 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:56.416736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.416698 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 16:43:56.416896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.416767 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 16:43:56.420683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.420653 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:56.656403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.655420 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:56.951832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.951752 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75"] Nov 03 16:43:56.957144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:56.957117 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.055232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.021584 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0ea8a5\"" Nov 03 16:43:57.055232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.025438 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0ea8a5-dockercfg-fhzjk\"" Nov 03 16:43:57.058168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.058130 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75"] Nov 03 16:43:57.155105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.154837 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e14ac8d-1359-4333-9cd5-098963a92698-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.155744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.155516 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdz5\" (UniqueName: \"kubernetes.io/projected/0e14ac8d-1359-4333-9cd5-098963a92698-kube-api-access-nmdz5\") pod \"payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.256755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.256724 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdz5\" (UniqueName: \"kubernetes.io/projected/0e14ac8d-1359-4333-9cd5-098963a92698-kube-api-access-nmdz5\") pod \"payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.256956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.256898 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e14ac8d-1359-4333-9cd5-098963a92698-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.257770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.257739 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e14ac8d-1359-4333-9cd5-098963a92698-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.317379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.317347 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdz5\" (UniqueName: \"kubernetes.io/projected/0e14ac8d-1359-4333-9cd5-098963a92698-kube-api-access-nmdz5\") pod \"payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.416934 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.416906 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:57.601527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.601428 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:57.801089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.801052 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:57.996516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.996486 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75"] Nov 03 16:43:57.998223 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:57.998186 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:57.998898 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:43:57.998867 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e14ac8d_1359_4333_9cd5_098963a92698.slice/crio-006ade6683b3ae54f76181750aa4285d7331ffa98cc958115bdf3aa3d9d37109 WatchSource:0}: Error finding container 006ade6683b3ae54f76181750aa4285d7331ffa98cc958115bdf3aa3d9d37109: Status 404 returned error can't find the container with id 006ade6683b3ae54f76181750aa4285d7331ffa98cc958115bdf3aa3d9d37109 Nov 03 16:43:58.062194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.062145 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n"] Nov 03 16:43:58.066860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.066831 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b0ac4d-ln49n"] Nov 03 16:43:58.197901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.197873 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:58.269788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.269691 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" event={"ID":"0e14ac8d-1359-4333-9cd5-098963a92698","Type":"ContainerStarted","Data":"1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963"} Nov 03 16:43:58.269788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.269745 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" event={"ID":"0e14ac8d-1359-4333-9cd5-098963a92698","Type":"ContainerStarted","Data":"006ade6683b3ae54f76181750aa4285d7331ffa98cc958115bdf3aa3d9d37109"} Nov 03 16:43:58.269788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.269767 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:43:58.270243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.270195 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:58.271286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.271257 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:43:58.393032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:58.392978 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" podStartSLOduration=2.392963946 podStartE2EDuration="2.392963946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:58.389372978 +0000 UTC m=+1908.510425815" watchObservedRunningTime="2026-11-03 16:43:58.392963946 +0000 UTC m=+1908.514016804" Nov 03 16:43:59.642983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:59.642931 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:59.910294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:43:59.904512 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:00.077234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:00.071575 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ad51b6-004e-458b-8851-764b3aae7795" path="/var/lib/kubelet/pods/64ad51b6-004e-458b-8851-764b3aae7795/volumes" Nov 03 16:44:00.118102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:00.116238 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:00.325317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:00.325280 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz"] Nov 03 16:44:00.738291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:00.733285 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:00.958071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:00.958037 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:01.157723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:01.157597 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:01.411841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:01.411693 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:03.355836 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:03.355776 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:05.164520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.164479 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:05.326152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.326114 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" podUID="9b44e7f3-e5a3-4ae1-b285-96f756a2b268" containerName="payload-pre-processing" containerID="cri-o://b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19" gracePeriod=25 Nov 03 16:44:05.448574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.448547 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:05.622747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.622721 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:44:05.734528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.734498 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-plugins-config-volume\") pod \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " Nov 03 16:44:05.734730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.734644 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-kube-api-access-6zqfx\" (UniqueName: \"kubernetes.io/projected/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-kube-api-access-6zqfx\") pod \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\" (UID: \"9b44e7f3-e5a3-4ae1-b285-96f756a2b268\") " Nov 03 16:44:05.734905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.734881 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-plugins-config-volume" pod "9b44e7f3-e5a3-4ae1-b285-96f756a2b268" (UID: "9b44e7f3-e5a3-4ae1-b285-96f756a2b268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:05.735309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.735289 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:05.736974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.736952 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-kube-api-access-6zqfx" pod "9b44e7f3-e5a3-4ae1-b285-96f756a2b268" (UID: "9b44e7f3-e5a3-4ae1-b285-96f756a2b268"). InnerVolumeSpecName "kube-api-access-6zqfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:05.836659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:05.836622 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqfx\" (UniqueName: \"kubernetes.io/projected/9b44e7f3-e5a3-4ae1-b285-96f756a2b268-kube-api-access-6zqfx\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:06.333854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.333764 5946 generic.go:356] "Generic (PLEG): container finished" podID="9b44e7f3-e5a3-4ae1-b285-96f756a2b268" containerID="b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19" exitCode=0 Nov 03 16:44:06.333854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.333812 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" Nov 03 16:44:06.334411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.333850 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" event={"ID":"9b44e7f3-e5a3-4ae1-b285-96f756a2b268","Type":"ContainerDied","Data":"b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19"} Nov 03 16:44:06.334411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.333920 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz" event={"ID":"9b44e7f3-e5a3-4ae1-b285-96f756a2b268","Type":"ContainerDied","Data":"f0a3507423b4f4bc0ca1dc1b6ef8c6e10b716f730d5453b5fa63dc1963ea6d85"} Nov 03 16:44:06.334411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.333953 5946 scope.go:122] "RemoveContainer" containerID="b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19" Nov 03 16:44:06.344425 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.344401 5946 scope.go:122] "RemoveContainer" containerID="b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19" Nov 03 16:44:06.344754 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:44:06.344730 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19\": container with ID starting with b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19 not found: ID does not exist" containerID="b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19" Nov 03 16:44:06.344865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.344758 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19"} err="failed to get container status \"b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19\": rpc error: code = NotFound desc = could not find container \"b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19\": container with ID starting with b89c365cd9fa04ff34495c56308a3cec3748d8b0e9f0b3761520a4c51648ae19 not found: ID does not exist" Nov 03 16:44:06.506448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.506412 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz"] Nov 03 16:44:06.557087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.557048 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9635db0d-bcd84d759-nblwz"] Nov 03 16:44:06.938177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:06.938143 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:07.123159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.123121 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:07.492991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.492947 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r"] Nov 03 16:44:07.503230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.495173 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9b44e7f3-e5a3-4ae1-b285-96f756a2b268" containerName="payload-pre-processing" Nov 03 16:44:07.558354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.558320 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b44e7f3-e5a3-4ae1-b285-96f756a2b268" containerName="payload-pre-processing" Nov 03 16:44:07.558916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.558894 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b44e7f3-e5a3-4ae1-b285-96f756a2b268" containerName="payload-pre-processing" Nov 03 16:44:07.626842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.626818 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r"] Nov 03 16:44:07.627056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.626928 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:07.682888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.682856 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6822cb\"" Nov 03 16:44:07.682888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.682861 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6822cb-dockercfg-8l9xh\"" Nov 03 16:44:07.723480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.723442 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkvt\" (UniqueName: \"kubernetes.io/projected/9135cfdb-7c1c-4a8b-833b-db32a0261825-kube-api-access-4wkvt\") pod \"payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:07.723727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.723703 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9135cfdb-7c1c-4a8b-833b-db32a0261825-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:07.825718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.825625 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkvt\" (UniqueName: \"kubernetes.io/projected/9135cfdb-7c1c-4a8b-833b-db32a0261825-kube-api-access-4wkvt\") pod \"payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:07.825903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.825834 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9135cfdb-7c1c-4a8b-833b-db32a0261825-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:07.826677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.826653 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9135cfdb-7c1c-4a8b-833b-db32a0261825-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:07.888908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.888875 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkvt\" (UniqueName: \"kubernetes.io/projected/9135cfdb-7c1c-4a8b-833b-db32a0261825-kube-api-access-4wkvt\") pod \"payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:07.949606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:07.949558 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:08.006781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:08.006745 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b44e7f3-e5a3-4ae1-b285-96f756a2b268" path="/var/lib/kubelet/pods/9b44e7f3-e5a3-4ae1-b285-96f756a2b268/volumes" Nov 03 16:44:08.361169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:08.361144 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r"] Nov 03 16:44:08.362138 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:44:08.362112 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9135cfdb_7c1c_4a8b_833b_db32a0261825.slice/crio-84285c5c34b904d87a4c2c55c149b8bc0d1c9df1b1a5593d783f3a21b8994028 WatchSource:0}: Error finding container 84285c5c34b904d87a4c2c55c149b8bc0d1c9df1b1a5593d783f3a21b8994028: Status 404 returned error can't find the container with id 84285c5c34b904d87a4c2c55c149b8bc0d1c9df1b1a5593d783f3a21b8994028 Nov 03 16:44:08.852667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:08.852636 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:09.192908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.192826 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl"] Nov 03 16:44:09.197027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.197007 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.245775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.245745 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.245931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.245904 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.245979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.245950 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwx8p\" (UniqueName: \"kubernetes.io/projected/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-kube-api-access-qwx8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.257652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.257621 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl"] Nov 03 16:44:09.280676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.280643 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:44:09.304077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.304042 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:44:09.304077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.304049 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:44:09.304308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.304092 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:44:09.304308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.304110 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:44:09.347396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.347362 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.347556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.347527 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.347600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.347563 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwx8p\" (UniqueName: \"kubernetes.io/projected/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-kube-api-access-qwx8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.348126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.348107 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.350056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.350037 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.358908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.358871 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" event={"ID":"9135cfdb-7c1c-4a8b-833b-db32a0261825","Type":"ContainerStarted","Data":"95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced"} Nov 03 16:44:09.359033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.358929 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:09.359033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.358945 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" event={"ID":"9135cfdb-7c1c-4a8b-833b-db32a0261825","Type":"ContainerStarted","Data":"84285c5c34b904d87a4c2c55c149b8bc0d1c9df1b1a5593d783f3a21b8994028"} Nov 03 16:44:09.408766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.408732 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwx8p\" (UniqueName: \"kubernetes.io/projected/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-kube-api-access-qwx8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.511751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.511720 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:09.564595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.564515 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" podStartSLOduration=2.564476024 podStartE2EDuration="2.564476024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:09.558939841 +0000 UTC m=+1919.679992664" watchObservedRunningTime="2026-11-03 16:44:09.564476024 +0000 UTC m=+1919.685528848" Nov 03 16:44:09.904921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:09.904890 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl"] Nov 03 16:44:09.905583 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:44:09.905559 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d11b7fc_8ad4_403e_90c9_2c1087b4a9f4.slice/crio-6e3e64e54822d625f0334c2562163f1bf1c4c47be98e40a14fcf9cb3ea68dd72 WatchSource:0}: Error finding container 6e3e64e54822d625f0334c2562163f1bf1c4c47be98e40a14fcf9cb3ea68dd72: Status 404 returned error can't find the container with id 6e3e64e54822d625f0334c2562163f1bf1c4c47be98e40a14fcf9cb3ea68dd72 Nov 03 16:44:10.368112 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:10.368074 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" event={"ID":"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4","Type":"ContainerStarted","Data":"a20a0eefdb361f8bf148bb160693374fcc5000d971df5e7b12c229e16744391b"} Nov 03 16:44:10.368281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:10.368141 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" event={"ID":"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4","Type":"ContainerStarted","Data":"6e3e64e54822d625f0334c2562163f1bf1c4c47be98e40a14fcf9cb3ea68dd72"} Nov 03 16:44:10.491830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:10.491716 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" podStartSLOduration=1.491703491 podStartE2EDuration="1.491703491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:10.48839584 +0000 UTC m=+1920.609448663" watchObservedRunningTime="2026-11-03 16:44:10.491703491 +0000 UTC m=+1920.612756313" Nov 03 16:44:11.378052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:11.377960 5946 generic.go:356] "Generic (PLEG): container finished" podID="8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" containerID="a20a0eefdb361f8bf148bb160693374fcc5000d971df5e7b12c229e16744391b" exitCode=0 Nov 03 16:44:11.378052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:11.378033 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" event={"ID":"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4","Type":"ContainerDied","Data":"a20a0eefdb361f8bf148bb160693374fcc5000d971df5e7b12c229e16744391b"} Nov 03 16:44:11.562006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:11.561974 5946 scope.go:122] "RemoveContainer" containerID="5ab48f6601eaf6273097249ea28c010998a0e339f2747067e3f8266eba25f924" Nov 03 16:44:11.572366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:11.572344 5946 scope.go:122] "RemoveContainer" containerID="031c2f63db4e41209036001a349781a8d1f48e890c7fcf880886976d254fa1de" Nov 03 16:44:12.553767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.553742 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:12.586952 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:44:12.586920 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:44:12.593585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.593565 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-kube-api-access-qwx8p\" (UniqueName: \"kubernetes.io/projected/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-kube-api-access-qwx8p\") pod \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " Nov 03 16:44:12.593708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.593694 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-openshift-ca\") pod \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " Nov 03 16:44:12.593753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.593724 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-xks-ca\") pod \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\" (UID: \"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4\") " Nov 03 16:44:12.594084 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.594056 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-openshift-ca" pod "8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" (UID: "8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:12.594414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.594395 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:12.595801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.595775 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-kube-api-access-qwx8p" pod "8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" (UID: "8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4"). InnerVolumeSpecName "kube-api-access-qwx8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:12.595922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.595906 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-xks-ca" pod "8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" (UID: "8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:12.695818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.695740 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwx8p\" (UniqueName: \"kubernetes.io/projected/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-kube-api-access-qwx8p\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:12.695818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:12.695779 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:13.339677 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:13.339404 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:13.394150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:13.394115 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" event={"ID":"8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4","Type":"ContainerDied","Data":"6e3e64e54822d625f0334c2562163f1bf1c4c47be98e40a14fcf9cb3ea68dd72"} Nov 03 16:44:13.394150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:13.394148 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl" Nov 03 16:44:13.394454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:13.394188 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3e64e54822d625f0334c2562163f1bf1c4c47be98e40a14fcf9cb3ea68dd72" Nov 03 16:44:13.557854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:13.557823 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:14.127322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:14.127288 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:17.650470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:17.650431 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq"] Nov 03 16:44:17.650896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:17.650634 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" podUID="d4f56784-9ff3-4b95-866a-e9b309101063" containerName="discovery" containerID="cri-o://48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf" gracePeriod=30 Nov 03 16:44:17.942310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:17.942277 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:44:18.034679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.034650 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-ca-configmap\") pod \"d4f56784-9ff3-4b95-866a-e9b309101063\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " Nov 03 16:44:18.034875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.034703 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f56784-9ff3-4b95-866a-e9b309101063-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f56784-9ff3-4b95-866a-e9b309101063-local-certs\") pod \"d4f56784-9ff3-4b95-866a-e9b309101063\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " Nov 03 16:44:18.034875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.034749 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-istio-token\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-istio-token\") pod \"d4f56784-9ff3-4b95-866a-e9b309101063\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " Nov 03 16:44:18.034875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.034784 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-kube-api-access-fjqkj\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-kube-api-access-fjqkj\") pod \"d4f56784-9ff3-4b95-866a-e9b309101063\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " Nov 03 16:44:18.035057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.035036 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-kubeconfig\") pod \"d4f56784-9ff3-4b95-866a-e9b309101063\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " Nov 03 16:44:18.035115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.035078 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-ca-configmap" pod "d4f56784-9ff3-4b95-866a-e9b309101063" (UID: "d4f56784-9ff3-4b95-866a-e9b309101063"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:18.035115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.035092 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-cacerts\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-cacerts\") pod \"d4f56784-9ff3-4b95-866a-e9b309101063\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " Nov 03 16:44:18.035268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.035125 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-dns-cert\") pod \"d4f56784-9ff3-4b95-866a-e9b309101063\" (UID: \"d4f56784-9ff3-4b95-866a-e9b309101063\") " Nov 03 16:44:18.035947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.035921 5946 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-ca-configmap\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:18.037448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.037420 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f56784-9ff3-4b95-866a-e9b309101063-local-certs" pod "d4f56784-9ff3-4b95-866a-e9b309101063" (UID: "d4f56784-9ff3-4b95-866a-e9b309101063"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:18.037913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.037878 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-kube-api-access-fjqkj" pod "d4f56784-9ff3-4b95-866a-e9b309101063" (UID: "d4f56784-9ff3-4b95-866a-e9b309101063"). InnerVolumeSpecName "kube-api-access-fjqkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:18.038024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.037971 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-cacerts" pod "d4f56784-9ff3-4b95-866a-e9b309101063" (UID: "d4f56784-9ff3-4b95-866a-e9b309101063"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:18.038024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.037993 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-istio-token" pod "d4f56784-9ff3-4b95-866a-e9b309101063" (UID: "d4f56784-9ff3-4b95-866a-e9b309101063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:18.038024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.037995 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-kubeconfig" pod "d4f56784-9ff3-4b95-866a-e9b309101063" (UID: "d4f56784-9ff3-4b95-866a-e9b309101063"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:18.038333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.038310 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-dns-cert" pod "d4f56784-9ff3-4b95-866a-e9b309101063" (UID: "d4f56784-9ff3-4b95-866a-e9b309101063"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:18.138245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.138186 5946 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f56784-9ff3-4b95-866a-e9b309101063-local-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:18.138245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.138244 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:18.138449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.138262 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqkj\" (UniqueName: \"kubernetes.io/projected/d4f56784-9ff3-4b95-866a-e9b309101063-kube-api-access-fjqkj\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:18.138449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.138280 5946 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-kubeconfig\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:18.138449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.138291 5946 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-cacerts\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:18.138449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.138299 5946 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4f56784-9ff3-4b95-866a-e9b309101063-istio-csr-dns-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:18.266326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.266294 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:18.435240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.435185 5946 generic.go:356] "Generic (PLEG): container finished" podID="d4f56784-9ff3-4b95-866a-e9b309101063" containerID="48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf" exitCode=0 Nov 03 16:44:18.435417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.435260 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" Nov 03 16:44:18.435417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.435271 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" event={"ID":"d4f56784-9ff3-4b95-866a-e9b309101063","Type":"ContainerDied","Data":"48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf"} Nov 03 16:44:18.435417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.435326 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq" event={"ID":"d4f56784-9ff3-4b95-866a-e9b309101063","Type":"ContainerDied","Data":"7d0d84792dc5ca5bd9b6c7b334dfea76d73eed51dd9144e0b8432d38fd2aa386"} Nov 03 16:44:18.435417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.435347 5946 scope.go:122] "RemoveContainer" containerID="48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf" Nov 03 16:44:18.447177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.447160 5946 scope.go:122] "RemoveContainer" containerID="48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf" Nov 03 16:44:18.447470 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:44:18.447454 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf\": container with ID starting with 48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf not found: ID does not exist" containerID="48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf" Nov 03 16:44:18.447520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.447477 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf"} err="failed to get container status \"48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf\": rpc error: code = NotFound desc = could not find container \"48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf\": container with ID starting with 48460fbcbb4e47047da6bc414fa37cf67775d3eed3478f63079ad5d4e6fd3dcf not found: ID does not exist" Nov 03 16:44:18.547147 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.547067 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:18.612788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.612756 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq"] Nov 03 16:44:18.618577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:18.618550 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b94nq"] Nov 03 16:44:19.302629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:19.302595 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:20.006660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:20.006620 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f56784-9ff3-4b95-866a-e9b309101063" path="/var/lib/kubelet/pods/d4f56784-9ff3-4b95-866a-e9b309101063/volumes" Nov 03 16:44:20.372009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:20.371918 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:44:20.988649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:20.988614 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl"] Nov 03 16:44:20.988828 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:20.988808 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" containerID="cri-o://acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d" gracePeriod=30 Nov 03 16:44:22.266261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:22.266184 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:22.355358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:22.355324 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4"] Nov 03 16:44:22.355717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:22.355688 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" containerID="cri-o://42154addac6e7a0815008ff21cec274dcba5ac2d33ba5dea418d8d425420ce27" gracePeriod=30 Nov 03 16:44:22.509350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:22.509236 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:24.345923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:24.345886 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:24.390130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:24.390096 5946 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:44:24.390389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:24.390170 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:44:24.888650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:24.888617 5946 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:44:24.888813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:24.888685 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:44:26.273398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.273335 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:44:26.368505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368469 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-credential-socket\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.368676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368523 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-istio-token\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-istio-token\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.368676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368581 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-data\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.368676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368628 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-envoy\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.368856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368689 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-certs\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.368856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368740 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61f27a78-889f-44c6-9e48-54ce4973190a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f27a78-889f-44c6-9e48-54ce4973190a-istio-podinfo\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.368856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368800 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f27a78-889f-44c6-9e48-54ce4973190a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f27a78-889f-44c6-9e48-54ce4973190a-istiod-ca-cert\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.368856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368816 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-credential-socket" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:26.368856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368837 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-socket\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.369140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368941 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-data" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:26.369140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368945 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-kube-api-access-qwrv2\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-kube-api-access-qwrv2\") pod \"61f27a78-889f-44c6-9e48-54ce4973190a\" (UID: \"61f27a78-889f-44c6-9e48-54ce4973190a\") " Nov 03 16:44:26.369140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.368983 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-certs" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:26.369318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.369193 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-socket" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:26.369318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.369280 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f27a78-889f-44c6-9e48-54ce4973190a-istiod-ca-cert" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:26.369706 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.369680 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.369706 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.369707 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.369887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.369717 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.369887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.369725 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f27a78-889f-44c6-9e48-54ce4973190a-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.369887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.369734 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.371153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.371124 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f27a78-889f-44c6-9e48-54ce4973190a-istio-podinfo" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:44:26.371153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.371139 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-istio-token" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:26.371517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.371495 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-envoy" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:26.371581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.371529 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-kube-api-access-qwrv2" pod "61f27a78-889f-44c6-9e48-54ce4973190a" (UID: "61f27a78-889f-44c6-9e48-54ce4973190a"). InnerVolumeSpecName "kube-api-access-qwrv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:26.471066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.471031 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f27a78-889f-44c6-9e48-54ce4973190a-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.471066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.471059 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f27a78-889f-44c6-9e48-54ce4973190a-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.471066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.471070 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrv2\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-kube-api-access-qwrv2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.471342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.471080 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f27a78-889f-44c6-9e48-54ce4973190a-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:26.539244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.539119 5946 generic.go:356] "Generic (PLEG): container finished" podID="61f27a78-889f-44c6-9e48-54ce4973190a" containerID="acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d" exitCode=0 Nov 03 16:44:26.539244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.539175 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" Nov 03 16:44:26.539244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.539227 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" event={"ID":"61f27a78-889f-44c6-9e48-54ce4973190a","Type":"ContainerDied","Data":"acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d"} Nov 03 16:44:26.539556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.539291 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl" event={"ID":"61f27a78-889f-44c6-9e48-54ce4973190a","Type":"ContainerDied","Data":"34e8b8f1a1e9c0076e1366605d5d58884f9b198ae985d01b4d48c4c55f7cc809"} Nov 03 16:44:26.539556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.539319 5946 scope.go:122] "RemoveContainer" containerID="acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d" Nov 03 16:44:26.551631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.551612 5946 scope.go:122] "RemoveContainer" containerID="acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d" Nov 03 16:44:26.551927 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:44:26.551902 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d\": container with ID starting with acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d not found: ID does not exist" containerID="acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d" Nov 03 16:44:26.552028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.551937 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d"} err="failed to get container status \"acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d\": rpc error: code = NotFound desc = could not find container \"acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d\": container with ID starting with acac6afc05ce947a676d3182bded730c67e2795124ae966caee85c7aef87525d not found: ID does not exist" Nov 03 16:44:26.714793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.714758 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl"] Nov 03 16:44:26.721926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:26.721901 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-q7fdl"] Nov 03 16:44:27.550996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.550961 5946 generic.go:356] "Generic (PLEG): container finished" podID="40910515-e3aa-4382-8139-2fb17a68649f" containerID="42154addac6e7a0815008ff21cec274dcba5ac2d33ba5dea418d8d425420ce27" exitCode=0 Nov 03 16:44:27.551622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.551106 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" event={"ID":"40910515-e3aa-4382-8139-2fb17a68649f","Type":"ContainerDied","Data":"42154addac6e7a0815008ff21cec274dcba5ac2d33ba5dea418d8d425420ce27"} Nov 03 16:44:27.798922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.798901 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:44:27.925628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.925590 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40910515-e3aa-4382-8139-2fb17a68649f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40910515-e3aa-4382-8139-2fb17a68649f-istio-podinfo\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.925801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.925767 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-kube-api-access-hh7fp\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-kube-api-access-hh7fp\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.925892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.925867 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-data\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.925957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.925901 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40910515-e3aa-4382-8139-2fb17a68649f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40910515-e3aa-4382-8139-2fb17a68649f-istiod-ca-cert\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.926012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.925990 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-envoy\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.926070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926046 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-certs\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.926158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926136 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-socket\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.926251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926173 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-data" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:27.926251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926197 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-istio-token\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-istio-token\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.926347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926255 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-credential-socket\") pod \"40910515-e3aa-4382-8139-2fb17a68649f\" (UID: \"40910515-e3aa-4382-8139-2fb17a68649f\") " Nov 03 16:44:27.926459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926433 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-socket" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:27.926726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926688 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-certs" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:27.926813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926761 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40910515-e3aa-4382-8139-2fb17a68649f-istiod-ca-cert" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:27.926920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926844 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-credential-socket" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:27.927033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.926987 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:27.927033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.927011 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:27.927033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.927027 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:27.928150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.928121 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40910515-e3aa-4382-8139-2fb17a68649f-istio-podinfo" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:44:27.928287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.928267 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-kube-api-access-hh7fp" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "kube-api-access-hh7fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:27.928539 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.928511 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-istio-token" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:27.928728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:27.928712 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-envoy" pod "40910515-e3aa-4382-8139-2fb17a68649f" (UID: "40910515-e3aa-4382-8139-2fb17a68649f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:28.007627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.007592 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" path="/var/lib/kubelet/pods/61f27a78-889f-44c6-9e48-54ce4973190a/volumes" Nov 03 16:44:28.028892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.028828 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:28.028892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.028864 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:28.028892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.028881 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40910515-e3aa-4382-8139-2fb17a68649f-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:28.029063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.028896 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh7fp\" (UniqueName: \"kubernetes.io/projected/40910515-e3aa-4382-8139-2fb17a68649f-kube-api-access-hh7fp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:28.029063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.028912 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40910515-e3aa-4382-8139-2fb17a68649f-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:28.029063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.028928 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40910515-e3aa-4382-8139-2fb17a68649f-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:28.042774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.042744 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:44:28.564724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.564697 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" Nov 03 16:44:28.565231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.564728 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4" event={"ID":"40910515-e3aa-4382-8139-2fb17a68649f","Type":"ContainerDied","Data":"0195fc3a67cf618cd560307c5664c70f462352c705334abef57c7dc1885e77e7"} Nov 03 16:44:28.565231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.564785 5946 scope.go:122] "RemoveContainer" containerID="42154addac6e7a0815008ff21cec274dcba5ac2d33ba5dea418d8d425420ce27" Nov 03 16:44:28.739998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.739960 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4"] Nov 03 16:44:28.791034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:28.790998 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-k8db4"] Nov 03 16:44:29.259814 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:29.259772 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:30.005444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:30.005405 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40910515-e3aa-4382-8139-2fb17a68649f" path="/var/lib/kubelet/pods/40910515-e3aa-4382-8139-2fb17a68649f/volumes" Nov 03 16:44:33.335240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:33.333913 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:33.537232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:33.536630 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:34.162439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:34.161513 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:35.433740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:35.433684 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:38.643545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:38.643436 5946 generic.go:356] "Generic (PLEG): container finished" podID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerID="6354178bb1b7a787fc607209340afaa73ab32dc200e05d774d32ec5d30fe628f" exitCode=0 Nov 03 16:44:38.644053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:38.643530 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" event={"ID":"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a","Type":"ContainerDied","Data":"6354178bb1b7a787fc607209340afaa73ab32dc200e05d774d32ec5d30fe628f"} Nov 03 16:44:38.644121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:38.644062 5946 scope.go:122] "RemoveContainer" containerID="6354178bb1b7a787fc607209340afaa73ab32dc200e05d774d32ec5d30fe628f" Nov 03 16:44:38.645977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:38.645950 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:44:38.646060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:38.646019 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:44:38.646060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:38.646053 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:44:38.949715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:38.948232 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:39.654717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:39.654680 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" event={"ID":"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a","Type":"ContainerStarted","Data":"3c229fbe88978a18ebc66c5e30bf454e582b731b65c3308d968e57f3a2aaf482"} Nov 03 16:44:39.945676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:39.945590 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:44:39.946791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:39.946765 5946 patch_prober.go:29] interesting pod/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 16:44:39.946890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:39.946816 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 16:44:40.341160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:40.341126 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:44:40.945821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:40.945775 5946 patch_prober.go:29] interesting pod/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 16:44:40.946389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:40.945858 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 16:44:41.946166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:41.946127 5946 patch_prober.go:29] interesting pod/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 16:44:41.946610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:41.946181 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 16:44:42.585716 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:44:42.585678 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:44:42.946025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:42.945945 5946 patch_prober.go:29] interesting pod/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 16:44:42.946025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:42.945994 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 16:44:43.599579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:43.599542 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:43.802273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:43.802185 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:43.945817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:43.945723 5946 patch_prober.go:29] interesting pod/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 16:44:43.945817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:43.945791 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 16:44:44.111728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:44.111700 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:44.950983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:44.950948 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:44:44.951944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:44.951922 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:44:48.463474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:48.463412 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:51.374852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:51.374815 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p"] Nov 03 16:44:51.375345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:51.375031 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" containerID="cri-o://6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631" gracePeriod=30 Nov 03 16:44:51.479237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:51.479186 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:44:51.479536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:51.479503 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:44:53.177704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:53.177666 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:56.652146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.652072 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:44:56.774375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774341 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-credential-socket\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774393 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-certs\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774427 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-socket\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774450 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-token\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-token\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774508 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-data\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774583 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istiod-ca-cert\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774662 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-credential-socket" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:56.774791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774703 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-podinfo\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774742 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-socket" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:56.774791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774748 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-envoy\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.774791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774760 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-certs" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:56.775081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774806 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-data" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:56.775081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.774834 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-kube-api-access-dx5pl\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-kube-api-access-dx5pl\") pod \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\" (UID: \"8062d5e7-e26d-4a4f-9309-dc05bf1551b3\") " Nov 03 16:44:56.775081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.775007 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istiod-ca-cert" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:56.775852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.775803 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.775852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.775836 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.775852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.775851 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.776130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.775872 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.776130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.775887 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.776971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.776946 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-token" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:56.777076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.777006 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-envoy" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:56.777076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.777041 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-podinfo" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:44:56.777243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.777193 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-kube-api-access-dx5pl" pod "8062d5e7-e26d-4a4f-9309-dc05bf1551b3" (UID: "8062d5e7-e26d-4a4f-9309-dc05bf1551b3"). InnerVolumeSpecName "kube-api-access-dx5pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:56.795321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.795296 5946 generic.go:356] "Generic (PLEG): container finished" podID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerID="6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631" exitCode=0 Nov 03 16:44:56.795447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.795349 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" Nov 03 16:44:56.795447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.795382 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" event={"ID":"8062d5e7-e26d-4a4f-9309-dc05bf1551b3","Type":"ContainerDied","Data":"6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631"} Nov 03 16:44:56.795748 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.795458 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p" event={"ID":"8062d5e7-e26d-4a4f-9309-dc05bf1551b3","Type":"ContainerDied","Data":"64f351e7486c54314b879d3b7c1417eb870a3de83403e27dda70dae4bd767c2e"} Nov 03 16:44:56.795748 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.795493 5946 scope.go:122] "RemoveContainer" containerID="6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631" Nov 03 16:44:56.807330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.807312 5946 scope.go:122] "RemoveContainer" containerID="6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631" Nov 03 16:44:56.807611 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:44:56.807591 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631\": container with ID starting with 6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631 not found: ID does not exist" containerID="6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631" Nov 03 16:44:56.807684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.807620 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631"} err="failed to get container status \"6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631\": rpc error: code = NotFound desc = could not find container \"6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631\": container with ID starting with 6a3209db5ac9ff5bb4c1774ce67d5d7f748eb2d5878e18e715d69b8785a01631 not found: ID does not exist" Nov 03 16:44:56.877096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.877055 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.877096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.877081 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.877096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.877093 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.877096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.877103 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5pl\" (UniqueName: \"kubernetes.io/projected/8062d5e7-e26d-4a4f-9309-dc05bf1551b3-kube-api-access-dx5pl\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:44:56.944467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.944382 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p"] Nov 03 16:44:56.949622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:56.949593 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-z267p"] Nov 03 16:44:57.973826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:57.973781 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:58.007769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:44:58.007739 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" path="/var/lib/kubelet/pods/8062d5e7-e26d-4a4f-9309-dc05bf1551b3/volumes" Nov 03 16:45:00.169784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.169747 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr"] Nov 03 16:45:00.170393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170379 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d4f56784-9ff3-4b95-866a-e9b309101063" containerName="discovery" Nov 03 16:45:00.170450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170394 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f56784-9ff3-4b95-866a-e9b309101063" containerName="discovery" Nov 03 16:45:00.170450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170418 5946 cpu_manager.go:414] "containerMap: removing container" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" Nov 03 16:45:00.170450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170424 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" Nov 03 16:45:00.170450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170437 5946 cpu_manager.go:414] "containerMap: removing container" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" Nov 03 16:45:00.170450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170443 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170454 5946 cpu_manager.go:414] "containerMap: removing container" podUID="8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" containerName="revoke-keys" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170459 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" containerName="revoke-keys" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170469 5946 cpu_manager.go:414] "containerMap: removing container" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170473 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170558 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f56784-9ff3-4b95-866a-e9b309101063" containerName="discovery" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170570 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="40910515-e3aa-4382-8139-2fb17a68649f" containerName="istio-proxy" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170577 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="8062d5e7-e26d-4a4f-9309-dc05bf1551b3" containerName="istio-proxy" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170583 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" containerName="revoke-keys" Nov 03 16:45:00.170645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.170590 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f27a78-889f-44c6-9e48-54ce4973190a" containerName="istio-proxy" Nov 03 16:45:00.175163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.175142 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.206539 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.206511 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng"] Nov 03 16:45:00.210478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.210458 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9"] Nov 03 16:45:00.210612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.210597 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.214673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.214653 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf"] Nov 03 16:45:00.214782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.214767 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.219295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.219275 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz"] Nov 03 16:45:00.219420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.219401 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.223058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.223038 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f"] Nov 03 16:45:00.223195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.223178 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.227535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.227386 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d"] Nov 03 16:45:00.227535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.227461 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.235345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.235325 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78"] Nov 03 16:45:00.241424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.241398 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.245412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.245380 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.254873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.254393 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:45:00.254873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.254540 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:45:00.254873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.254618 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:45:00.261503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.261480 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw"] Nov 03 16:45:00.265815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.265795 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84"] Nov 03 16:45:00.265926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.265897 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.271141 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.271119 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb"] Nov 03 16:45:00.271298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.271169 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.276286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.276265 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7"] Nov 03 16:45:00.276404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.276355 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.281576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.281559 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r"] Nov 03 16:45:00.281678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.281663 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.285702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.285680 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9"] Nov 03 16:45:00.285810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.285724 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng"] Nov 03 16:45:00.285810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.285756 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr"] Nov 03 16:45:00.285810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.285772 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.288664 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.288643 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh"] Nov 03 16:45:00.292723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.292705 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.321407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.321369 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf"] Nov 03 16:45:00.325757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.325731 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz"] Nov 03 16:45:00.326006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.325982 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnlv\" (UniqueName: \"kubernetes.io/projected/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-kube-api-access-hpnlv\") pod \"maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.326065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326022 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de379cd-d26a-4e49-be70-f220543c177a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.326125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326110 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5v5\" (UniqueName: \"kubernetes.io/projected/7101b7e0-dfa3-4b10-b60b-e515c861f40e-kube-api-access-qw5v5\") pod \"maas-api-key-cleanup-29895405-gpl78\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.326267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326244 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a9eb-4be2-4031-822d-b49a4670b679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.326382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326363 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.326459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326411 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g55ps\" (UniqueName: \"kubernetes.io/projected/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-kube-api-access-g55ps\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.326459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326452 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d096c2-13d5-417c-a2d8-498581ab250f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.326576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326487 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2qm\" (UniqueName: \"kubernetes.io/projected/65d096c2-13d5-417c-a2d8-498581ab250f-kube-api-access-bv2qm\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.326634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326537 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566f8\" (UniqueName: \"kubernetes.io/projected/e82edb20-138b-4058-8303-e6b1655bc37e-kube-api-access-566f8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.326684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326640 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82edb20-138b-4058-8303-e6b1655bc37e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.326738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326716 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqbs\" (UniqueName: \"kubernetes.io/projected/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-kube-api-access-fkqbs\") pod \"maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.326801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326739 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw89r\" (UniqueName: \"kubernetes.io/projected/4cd2a9eb-4be2-4031-822d-b49a4670b679-kube-api-access-mw89r\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.326801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326777 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.326904 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326878 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.326977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326957 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7101b7e0-dfa3-4b10-b60b-e515c861f40e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-gpl78\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.327050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.326998 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4smb\" (UniqueName: \"kubernetes.io/projected/3de379cd-d26a-4e49-be70-f220543c177a-kube-api-access-f4smb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.333530 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.333275 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78"] Nov 03 16:45:00.337373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.336258 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw"] Nov 03 16:45:00.377437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.376874 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7"] Nov 03 16:45:00.382171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.382141 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb"] Nov 03 16:45:00.408176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.408139 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh"] Nov 03 16:45:00.411687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.411658 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d"] Nov 03 16:45:00.414897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.414864 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84"] Nov 03 16:45:00.417449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.417422 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f"] Nov 03 16:45:00.428906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.428834 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.428906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.428875 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba75d816-f403-4721-acc8-60bddc30be52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.428906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.428896 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gpf\" (UniqueName: \"kubernetes.io/projected/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-kube-api-access-s8gpf\") pod \"maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.429146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.428918 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g55ps\" (UniqueName: \"kubernetes.io/projected/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-kube-api-access-g55ps\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.429146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.428946 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d096c2-13d5-417c-a2d8-498581ab250f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.429146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429032 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2qm\" (UniqueName: \"kubernetes.io/projected/65d096c2-13d5-417c-a2d8-498581ab250f-kube-api-access-bv2qm\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.429146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429091 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566f8\" (UniqueName: \"kubernetes.io/projected/e82edb20-138b-4058-8303-e6b1655bc37e-kube-api-access-566f8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.429146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429122 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpnb\" (UniqueName: \"kubernetes.io/projected/b5836760-c55b-46d4-9050-f33870943c0b-kube-api-access-wwpnb\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.429492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429160 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82edb20-138b-4058-8303-e6b1655bc37e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.429492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429287 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrf6\" (UniqueName: \"kubernetes.io/projected/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-kube-api-access-wmrf6\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.429492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429340 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70db9170-6a1c-4b21-b509-67d85e1cee91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.429492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429434 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.429722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429536 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqbs\" (UniqueName: \"kubernetes.io/projected/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-kube-api-access-fkqbs\") pod \"maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.429722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429573 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw89r\" (UniqueName: \"kubernetes.io/projected/4cd2a9eb-4be2-4031-822d-b49a4670b679-kube-api-access-mw89r\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.429722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429615 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.429722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429621 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.429722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429645 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.429722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429688 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvnz\" (UniqueName: \"kubernetes.io/projected/70db9170-6a1c-4b21-b509-67d85e1cee91-kube-api-access-8lvnz\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.429913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429783 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d096c2-13d5-417c-a2d8-498581ab250f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.429913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429838 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5836760-c55b-46d4-9050-f33870943c0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.429913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429875 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.429913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429906 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtf8\" (UniqueName: \"kubernetes.io/projected/ba75d816-f403-4721-acc8-60bddc30be52-kube-api-access-jhtf8\") pod \"maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.430037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429937 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82edb20-138b-4058-8303-e6b1655bc37e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.430037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.429947 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7101b7e0-dfa3-4b10-b60b-e515c861f40e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-gpl78\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.430129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430074 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4smb\" (UniqueName: \"kubernetes.io/projected/3de379cd-d26a-4e49-be70-f220543c177a-kube-api-access-f4smb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.430185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430141 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnlv\" (UniqueName: \"kubernetes.io/projected/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-kube-api-access-hpnlv\") pod \"maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.430185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430168 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de379cd-d26a-4e49-be70-f220543c177a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.430306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430238 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.430306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430291 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5v5\" (UniqueName: \"kubernetes.io/projected/7101b7e0-dfa3-4b10-b60b-e515c861f40e-kube-api-access-qw5v5\") pod \"maas-api-key-cleanup-29895405-gpl78\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.430411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430323 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a9eb-4be2-4031-822d-b49a4670b679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.430411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430353 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sth\" (UniqueName: \"kubernetes.io/projected/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-kube-api-access-w2sth\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.430526 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430434 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.430736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430707 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.430736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430711 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7101b7e0-dfa3-4b10-b60b-e515c861f40e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-gpl78\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.430937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.430896 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de379cd-d26a-4e49-be70-f220543c177a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.431026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.431007 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a9eb-4be2-4031-822d-b49a4670b679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.445075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.445050 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r"] Nov 03 16:45:00.491657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.491626 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5v5\" (UniqueName: \"kubernetes.io/projected/7101b7e0-dfa3-4b10-b60b-e515c861f40e-kube-api-access-qw5v5\") pod \"maas-api-key-cleanup-29895405-gpl78\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.492280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.492247 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566f8\" (UniqueName: \"kubernetes.io/projected/e82edb20-138b-4058-8303-e6b1655bc37e-kube-api-access-566f8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.493028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.493001 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2qm\" (UniqueName: \"kubernetes.io/projected/65d096c2-13d5-417c-a2d8-498581ab250f-kube-api-access-bv2qm\") pod \"maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.493139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.493003 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4smb\" (UniqueName: \"kubernetes.io/projected/3de379cd-d26a-4e49-be70-f220543c177a-kube-api-access-f4smb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.493139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.493043 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqbs\" (UniqueName: \"kubernetes.io/projected/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-kube-api-access-fkqbs\") pod \"maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.493321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.493225 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnlv\" (UniqueName: \"kubernetes.io/projected/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-kube-api-access-hpnlv\") pod \"maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.494253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.494191 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw89r\" (UniqueName: \"kubernetes.io/projected/4cd2a9eb-4be2-4031-822d-b49a4670b679-kube-api-access-mw89r\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.494448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.494427 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55ps\" (UniqueName: \"kubernetes.io/projected/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-kube-api-access-g55ps\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.527517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.527484 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:00.532316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532290 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.532437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532349 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sth\" (UniqueName: \"kubernetes.io/projected/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-kube-api-access-w2sth\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.532437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532406 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba75d816-f403-4721-acc8-60bddc30be52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.532940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532635 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gpf\" (UniqueName: \"kubernetes.io/projected/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-kube-api-access-s8gpf\") pod \"maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.532940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532726 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpnb\" (UniqueName: \"kubernetes.io/projected/b5836760-c55b-46d4-9050-f33870943c0b-kube-api-access-wwpnb\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.532940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532772 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrf6\" (UniqueName: \"kubernetes.io/projected/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-kube-api-access-wmrf6\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.532940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532814 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70db9170-6a1c-4b21-b509-67d85e1cee91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.532940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532860 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.532940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532915 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.533308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.532962 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvnz\" (UniqueName: \"kubernetes.io/projected/70db9170-6a1c-4b21-b509-67d85e1cee91-kube-api-access-8lvnz\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.533308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533041 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5836760-c55b-46d4-9050-f33870943c0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.533308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533095 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba75d816-f403-4721-acc8-60bddc30be52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.533308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533103 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtf8\" (UniqueName: \"kubernetes.io/projected/ba75d816-f403-4721-acc8-60bddc30be52-kube-api-access-jhtf8\") pod \"maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.533308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533098 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.533828 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533789 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70db9170-6a1c-4b21-b509-67d85e1cee91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.533914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533828 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.533998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533980 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.534050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.533987 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5836760-c55b-46d4-9050-f33870943c0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.548541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.548516 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:00.558782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.558756 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:00.569442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.569416 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:00.581447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.581419 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:00.588168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.587972 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:00.597378 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.595865 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:00.598085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.597920 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gpf\" (UniqueName: \"kubernetes.io/projected/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-kube-api-access-s8gpf\") pod \"maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.602993 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.602967 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:00.604034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.604003 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtf8\" (UniqueName: \"kubernetes.io/projected/ba75d816-f403-4721-acc8-60bddc30be52-kube-api-access-jhtf8\") pod \"maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.604170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.604144 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrf6\" (UniqueName: \"kubernetes.io/projected/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-kube-api-access-wmrf6\") pod \"maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.608837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.608186 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpnb\" (UniqueName: \"kubernetes.io/projected/b5836760-c55b-46d4-9050-f33870943c0b-kube-api-access-wwpnb\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.608837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.608766 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sth\" (UniqueName: \"kubernetes.io/projected/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-kube-api-access-w2sth\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.611611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.611371 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvnz\" (UniqueName: \"kubernetes.io/projected/70db9170-6a1c-4b21-b509-67d85e1cee91-kube-api-access-8lvnz\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.612283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.612025 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:00.635291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.633338 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:00.644664 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.644595 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:00.650333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.650052 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:00.658544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.658502 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:00.791431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.791189 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:00.930284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:00.930153 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng"] Nov 03 16:45:01.164318 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.164277 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82edb20_138b_4058_8303_e6b1655bc37e.slice/crio-305c3cd4d27cb855a34458463aae654c2b31905a61bd4673e07e6bba2a5921bb WatchSource:0}: Error finding container 305c3cd4d27cb855a34458463aae654c2b31905a61bd4673e07e6bba2a5921bb: Status 404 returned error can't find the container with id 305c3cd4d27cb855a34458463aae654c2b31905a61bd4673e07e6bba2a5921bb Nov 03 16:45:01.165657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.165536 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9"] Nov 03 16:45:01.359433 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.359403 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d27bf2c_ea47_4a96_931c_bbf2c2ba0d44.slice/crio-2af1048226c704c6bd06c4226bf45231f8dc2498f814a98c68075409bf683fed WatchSource:0}: Error finding container 2af1048226c704c6bd06c4226bf45231f8dc2498f814a98c68075409bf683fed: Status 404 returned error can't find the container with id 2af1048226c704c6bd06c4226bf45231f8dc2498f814a98c68075409bf683fed Nov 03 16:45:01.359753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.359532 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw"] Nov 03 16:45:01.380306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.380279 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84"] Nov 03 16:45:01.380571 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.380548 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2f4ad1_baad_41c9_a3e4_923dcae51c41.slice/crio-81bf33e8d4be82235b91439e0b9a6d92d2aafb1f7e15957d6a032647c0135045 WatchSource:0}: Error finding container 81bf33e8d4be82235b91439e0b9a6d92d2aafb1f7e15957d6a032647c0135045: Status 404 returned error can't find the container with id 81bf33e8d4be82235b91439e0b9a6d92d2aafb1f7e15957d6a032647c0135045 Nov 03 16:45:01.387822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.387791 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r"] Nov 03 16:45:01.393513 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.393491 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefd3d1f_d7e6_4bef_b033_508b910c0a2d.slice/crio-a43f960331f477a4c6811c9c0b1e2e8c34fe66d84a3d6d1339e6f69869dbf447 WatchSource:0}: Error finding container a43f960331f477a4c6811c9c0b1e2e8c34fe66d84a3d6d1339e6f69869dbf447: Status 404 returned error can't find the container with id a43f960331f477a4c6811c9c0b1e2e8c34fe66d84a3d6d1339e6f69869dbf447 Nov 03 16:45:01.409037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.409012 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf"] Nov 03 16:45:01.417052 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.416012 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd2a9eb_4be2_4031_822d_b49a4670b679.slice/crio-03112f578453e7ecff243d7ded12c207b5b82475ec7826e8934b5b66b52d85be WatchSource:0}: Error finding container 03112f578453e7ecff243d7ded12c207b5b82475ec7826e8934b5b66b52d85be: Status 404 returned error can't find the container with id 03112f578453e7ecff243d7ded12c207b5b82475ec7826e8934b5b66b52d85be Nov 03 16:45:01.417326 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.417305 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5836760_c55b_46d4_9050_f33870943c0b.slice/crio-4b17f043a63ae58e2804cb42e59e9830023f09e31badf3ed205d7fa552e8ea94 WatchSource:0}: Error finding container 4b17f043a63ae58e2804cb42e59e9830023f09e31badf3ed205d7fa552e8ea94: Status 404 returned error can't find the container with id 4b17f043a63ae58e2804cb42e59e9830023f09e31badf3ed205d7fa552e8ea94 Nov 03 16:45:01.419980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.418495 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb"] Nov 03 16:45:01.464041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.464015 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d"] Nov 03 16:45:01.464550 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.464528 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbfe05c_7d59_4e1a_81ed_f480be5892d0.slice/crio-942eae2fa096fb2531ba9ac247197a8b306ca6034801b6798b745191f7ec6e4e WatchSource:0}: Error finding container 942eae2fa096fb2531ba9ac247197a8b306ca6034801b6798b745191f7ec6e4e: Status 404 returned error can't find the container with id 942eae2fa096fb2531ba9ac247197a8b306ca6034801b6798b745191f7ec6e4e Nov 03 16:45:01.465736 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.465712 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d096c2_13d5_417c_a2d8_498581ab250f.slice/crio-b2a6cb0ab6b8efb27ed462e1a9d973e43270171ed2c2e2d407d06d2b19f45d50 WatchSource:0}: Error finding container b2a6cb0ab6b8efb27ed462e1a9d973e43270171ed2c2e2d407d06d2b19f45d50: Status 404 returned error can't find the container with id b2a6cb0ab6b8efb27ed462e1a9d973e43270171ed2c2e2d407d06d2b19f45d50 Nov 03 16:45:01.468562 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.468532 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f"] Nov 03 16:45:01.481992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.481967 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz"] Nov 03 16:45:01.491983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.491961 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78"] Nov 03 16:45:01.495637 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.495607 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7101b7e0_dfa3_4b10_b60b_e515c861f40e.slice/crio-ae0158fc86c6b4df5626f40f9b74c942a8b27cc410a4959c3fe6efbb8ae11e5b WatchSource:0}: Error finding container ae0158fc86c6b4df5626f40f9b74c942a8b27cc410a4959c3fe6efbb8ae11e5b: Status 404 returned error can't find the container with id ae0158fc86c6b4df5626f40f9b74c942a8b27cc410a4959c3fe6efbb8ae11e5b Nov 03 16:45:01.496929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.496908 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh"] Nov 03 16:45:01.498852 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.498820 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba75d816_f403_4721_acc8_60bddc30be52.slice/crio-e36014e90abdfc484f1f6ddf7139ba5a514b56a1070f86a97553e72ed7cb6e3f WatchSource:0}: Error finding container e36014e90abdfc484f1f6ddf7139ba5a514b56a1070f86a97553e72ed7cb6e3f: Status 404 returned error can't find the container with id e36014e90abdfc484f1f6ddf7139ba5a514b56a1070f86a97553e72ed7cb6e3f Nov 03 16:45:01.500973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.500957 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr"] Nov 03 16:45:01.503331 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.503311 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70db9170_6a1c_4b21_b509_67d85e1cee91.slice/crio-aaf6d4a0228fdcfbccd7d33c096d250a67453b3ea5f24dece8876cea63b070ee WatchSource:0}: Error finding container aaf6d4a0228fdcfbccd7d33c096d250a67453b3ea5f24dece8876cea63b070ee: Status 404 returned error can't find the container with id aaf6d4a0228fdcfbccd7d33c096d250a67453b3ea5f24dece8876cea63b070ee Nov 03 16:45:01.504598 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:01.504574 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3a99a6_3244_414d_aaba_1ac35a4ec7a5.slice/crio-f256651d11ca24947f5f931014de8c54861209af175b76c15f63dd38558e78f5 WatchSource:0}: Error finding container f256651d11ca24947f5f931014de8c54861209af175b76c15f63dd38558e78f5: Status 404 returned error can't find the container with id f256651d11ca24947f5f931014de8c54861209af175b76c15f63dd38558e78f5 Nov 03 16:45:01.505825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.505803 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7"] Nov 03 16:45:01.859639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.859606 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerStarted","Data":"2af1048226c704c6bd06c4226bf45231f8dc2498f814a98c68075409bf683fed"} Nov 03 16:45:01.860813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.860788 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerStarted","Data":"a43f960331f477a4c6811c9c0b1e2e8c34fe66d84a3d6d1339e6f69869dbf447"} Nov 03 16:45:01.861970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.861947 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerStarted","Data":"81bf33e8d4be82235b91439e0b9a6d92d2aafb1f7e15957d6a032647c0135045"} Nov 03 16:45:01.863088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.863070 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" event={"ID":"4cd2a9eb-4be2-4031-822d-b49a4670b679","Type":"ContainerStarted","Data":"03112f578453e7ecff243d7ded12c207b5b82475ec7826e8934b5b66b52d85be"} Nov 03 16:45:01.864178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.864159 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerStarted","Data":"ecb4d00fa2b50c30e0ff31178ef145e593c31811c14306090cccd1e138461cb7"} Nov 03 16:45:01.865301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.865282 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" event={"ID":"7101b7e0-dfa3-4b10-b60b-e515c861f40e","Type":"ContainerStarted","Data":"ae0158fc86c6b4df5626f40f9b74c942a8b27cc410a4959c3fe6efbb8ae11e5b"} Nov 03 16:45:01.866833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.866807 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" event={"ID":"e82edb20-138b-4058-8303-e6b1655bc37e","Type":"ContainerStarted","Data":"c14585da915cba467e7f6a657e2839b46d34bceb4080a89423300ce30519a2a6"} Nov 03 16:45:01.866936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.866851 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" event={"ID":"e82edb20-138b-4058-8303-e6b1655bc37e","Type":"ContainerStarted","Data":"305c3cd4d27cb855a34458463aae654c2b31905a61bd4673e07e6bba2a5921bb"} Nov 03 16:45:01.868177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.868153 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerStarted","Data":"b2a6cb0ab6b8efb27ed462e1a9d973e43270171ed2c2e2d407d06d2b19f45d50"} Nov 03 16:45:01.869503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.869483 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" event={"ID":"ba75d816-f403-4721-acc8-60bddc30be52","Type":"ContainerStarted","Data":"e36014e90abdfc484f1f6ddf7139ba5a514b56a1070f86a97553e72ed7cb6e3f"} Nov 03 16:45:01.870794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.870768 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerStarted","Data":"942eae2fa096fb2531ba9ac247197a8b306ca6034801b6798b745191f7ec6e4e"} Nov 03 16:45:01.872150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.872130 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" event={"ID":"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5","Type":"ContainerStarted","Data":"f256651d11ca24947f5f931014de8c54861209af175b76c15f63dd38558e78f5"} Nov 03 16:45:01.874123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.873750 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" event={"ID":"70db9170-6a1c-4b21-b509-67d85e1cee91","Type":"ContainerStarted","Data":"aaf6d4a0228fdcfbccd7d33c096d250a67453b3ea5f24dece8876cea63b070ee"} Nov 03 16:45:01.875682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.875659 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" event={"ID":"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7","Type":"ContainerStarted","Data":"e2c791d48c2716fdd7ae4aec3c4cbc52ec235d4e3a9a6b235df76245016bf3bd"} Nov 03 16:45:01.875772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.875711 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" event={"ID":"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7","Type":"ContainerStarted","Data":"4bdf6fe4bc671ce569b0277a8dd6c7bd0dc2a1382e52e8803731f3bd6943e893"} Nov 03 16:45:01.877026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:01.877007 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerStarted","Data":"4b17f043a63ae58e2804cb42e59e9830023f09e31badf3ed205d7fa552e8ea94"} Nov 03 16:45:02.152435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.152378 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" podStartSLOduration=2.15236341 podStartE2EDuration="2.15236341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.150607346 +0000 UTC m=+1972.271660169" watchObservedRunningTime="2026-11-03 16:45:02.15236341 +0000 UTC m=+1972.273416233" Nov 03 16:45:02.153331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.153298 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" podStartSLOduration=2.153290458 podStartE2EDuration="2.153290458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.026809192 +0000 UTC m=+1972.147862016" watchObservedRunningTime="2026-11-03 16:45:02.153290458 +0000 UTC m=+1972.274343281" Nov 03 16:45:02.675809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.675774 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:02.888796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.888689 5946 generic.go:356] "Generic (PLEG): container finished" podID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerID="e2c791d48c2716fdd7ae4aec3c4cbc52ec235d4e3a9a6b235df76245016bf3bd" exitCode=60 Nov 03 16:45:02.888796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.888770 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" event={"ID":"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7","Type":"ContainerDied","Data":"e2c791d48c2716fdd7ae4aec3c4cbc52ec235d4e3a9a6b235df76245016bf3bd"} Nov 03 16:45:02.889052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.889034 5946 scope.go:122] "RemoveContainer" containerID="e2c791d48c2716fdd7ae4aec3c4cbc52ec235d4e3a9a6b235df76245016bf3bd" Nov 03 16:45:02.890688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.890661 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerStarted","Data":"b69ef8a059d620dcd970e0dffd01039fb5c6db27a664b8d1870e24c3b5c55018"} Nov 03 16:45:02.892445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.892421 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerStarted","Data":"436f543620d0fd7f00902e37865badd21b254a51df1d27a147c4536a4f6de5b6"} Nov 03 16:45:02.895347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.895313 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerStarted","Data":"eda0dfde12e72dd39d63cee301ad6acadcf07c40c1d8670408ce16f857812b49"} Nov 03 16:45:02.895482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.895468 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerStarted","Data":"322e40faf4b7b8c7b3523f4ef4fed0c6dfdf330e8952ea330959e46fbadf7978"} Nov 03 16:45:02.897075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.897052 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" event={"ID":"4cd2a9eb-4be2-4031-822d-b49a4670b679","Type":"ContainerStarted","Data":"525f807c3f6ab4af37333927ca6e589f9fa2425b0ced0cd3b0e68864bcb81e16"} Nov 03 16:45:02.898797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.898770 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerStarted","Data":"db9b97fa4717c731388971ed63581a75990805a2640b2a64d7569cb2ad81ea84"} Nov 03 16:45:02.900543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.900503 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" event={"ID":"7101b7e0-dfa3-4b10-b60b-e515c861f40e","Type":"ContainerStarted","Data":"a68244fb013d6af5901013906a7f46d642cce366c8611abb3d3ca5905af19dfb"} Nov 03 16:45:02.902120 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.902098 5946 generic.go:356] "Generic (PLEG): container finished" podID="e82edb20-138b-4058-8303-e6b1655bc37e" containerID="c14585da915cba467e7f6a657e2839b46d34bceb4080a89423300ce30519a2a6" exitCode=60 Nov 03 16:45:02.902221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.902174 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" event={"ID":"e82edb20-138b-4058-8303-e6b1655bc37e","Type":"ContainerDied","Data":"c14585da915cba467e7f6a657e2839b46d34bceb4080a89423300ce30519a2a6"} Nov 03 16:45:02.902461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.902444 5946 scope.go:122] "RemoveContainer" containerID="c14585da915cba467e7f6a657e2839b46d34bceb4080a89423300ce30519a2a6" Nov 03 16:45:02.903907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.903884 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerStarted","Data":"816152a61dddc61e20fe4df62d40770d4a017dbd689989e6c8116cb78c83bc5d"} Nov 03 16:45:02.906049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.906028 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" event={"ID":"ba75d816-f403-4721-acc8-60bddc30be52","Type":"ContainerStarted","Data":"6c566db842b5bd2c0afe8315d9702cdd83bc225d7591eefee79ae569644e5ecc"} Nov 03 16:45:02.907842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.907821 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerStarted","Data":"07cd07e79d04987d7099b0674df3b99c0289b5ad0eb6f3369e4121331b1ab491"} Nov 03 16:45:02.922186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.917942 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" event={"ID":"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5","Type":"ContainerStarted","Data":"71839896a60a83a47393e5a80f8f8b2bed136cd50ec1f5a9dfdc8e7cc4ee503f"} Nov 03 16:45:02.925804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:02.925776 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" event={"ID":"70db9170-6a1c-4b21-b509-67d85e1cee91","Type":"ContainerStarted","Data":"ed5df2f6b1782a19180fb69d7bd4e71716c31d1c414e50ec6e75763477efc4dd"} Nov 03 16:45:03.135358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.135297 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" podStartSLOduration=3.135191283 podStartE2EDuration="3.135191283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.134121561 +0000 UTC m=+1973.255174397" watchObservedRunningTime="2026-11-03 16:45:03.135191283 +0000 UTC m=+1973.256244107" Nov 03 16:45:03.256577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.256514 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" podStartSLOduration=3.256499397 podStartE2EDuration="3.256499397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.254094342 +0000 UTC m=+1973.375147167" watchObservedRunningTime="2026-11-03 16:45:03.256499397 +0000 UTC m=+1973.377552220" Nov 03 16:45:03.389930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.389876 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" podStartSLOduration=3.3898641290000002 podStartE2EDuration="3.389864129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.388341559 +0000 UTC m=+1973.509394379" watchObservedRunningTime="2026-11-03 16:45:03.389864129 +0000 UTC m=+1973.510916993" Nov 03 16:45:03.514754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.514705 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" podStartSLOduration=3.514691961 podStartE2EDuration="3.514691961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.51104515 +0000 UTC m=+1973.632097976" watchObservedRunningTime="2026-11-03 16:45:03.514691961 +0000 UTC m=+1973.635744784" Nov 03 16:45:03.630314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.630257 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" podStartSLOduration=3.630244929 podStartE2EDuration="3.630244929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.628987251 +0000 UTC m=+1973.750040076" watchObservedRunningTime="2026-11-03 16:45:03.630244929 +0000 UTC m=+1973.751297772" Nov 03 16:45:03.867409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.867352 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" podStartSLOduration=3.8673344370000002 podStartE2EDuration="3.867334437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.864846051 +0000 UTC m=+1973.985898874" watchObservedRunningTime="2026-11-03 16:45:03.867334437 +0000 UTC m=+1973.988387262" Nov 03 16:45:03.935885 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.935855 5946 generic.go:356] "Generic (PLEG): container finished" podID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerID="07cd07e79d04987d7099b0674df3b99c0289b5ad0eb6f3369e4121331b1ab491" exitCode=60 Nov 03 16:45:03.936027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.935941 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerDied","Data":"07cd07e79d04987d7099b0674df3b99c0289b5ad0eb6f3369e4121331b1ab491"} Nov 03 16:45:03.936268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.936250 5946 scope.go:122] "RemoveContainer" containerID="07cd07e79d04987d7099b0674df3b99c0289b5ad0eb6f3369e4121331b1ab491" Nov 03 16:45:03.937684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.937661 5946 generic.go:356] "Generic (PLEG): container finished" podID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerID="71839896a60a83a47393e5a80f8f8b2bed136cd50ec1f5a9dfdc8e7cc4ee503f" exitCode=60 Nov 03 16:45:03.937795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.937735 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" event={"ID":"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5","Type":"ContainerDied","Data":"71839896a60a83a47393e5a80f8f8b2bed136cd50ec1f5a9dfdc8e7cc4ee503f"} Nov 03 16:45:03.937999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.937983 5946 scope.go:122] "RemoveContainer" containerID="71839896a60a83a47393e5a80f8f8b2bed136cd50ec1f5a9dfdc8e7cc4ee503f" Nov 03 16:45:03.939927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.939867 5946 generic.go:356] "Generic (PLEG): container finished" podID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerID="ed5df2f6b1782a19180fb69d7bd4e71716c31d1c414e50ec6e75763477efc4dd" exitCode=60 Nov 03 16:45:03.939990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.939930 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" event={"ID":"70db9170-6a1c-4b21-b509-67d85e1cee91","Type":"ContainerDied","Data":"ed5df2f6b1782a19180fb69d7bd4e71716c31d1c414e50ec6e75763477efc4dd"} Nov 03 16:45:03.940266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.940245 5946 scope.go:122] "RemoveContainer" containerID="ed5df2f6b1782a19180fb69d7bd4e71716c31d1c414e50ec6e75763477efc4dd" Nov 03 16:45:03.943585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.943565 5946 generic.go:356] "Generic (PLEG): container finished" podID="b5836760-c55b-46d4-9050-f33870943c0b" containerID="b69ef8a059d620dcd970e0dffd01039fb5c6db27a664b8d1870e24c3b5c55018" exitCode=60 Nov 03 16:45:03.943657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.943643 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerDied","Data":"b69ef8a059d620dcd970e0dffd01039fb5c6db27a664b8d1870e24c3b5c55018"} Nov 03 16:45:03.943816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.943804 5946 scope.go:122] "RemoveContainer" containerID="b69ef8a059d620dcd970e0dffd01039fb5c6db27a664b8d1870e24c3b5c55018" Nov 03 16:45:03.945254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.945199 5946 generic.go:356] "Generic (PLEG): container finished" podID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerID="436f543620d0fd7f00902e37865badd21b254a51df1d27a147c4536a4f6de5b6" exitCode=60 Nov 03 16:45:03.945345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.945325 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerDied","Data":"436f543620d0fd7f00902e37865badd21b254a51df1d27a147c4536a4f6de5b6"} Nov 03 16:45:03.945508 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.945491 5946 scope.go:122] "RemoveContainer" containerID="436f543620d0fd7f00902e37865badd21b254a51df1d27a147c4536a4f6de5b6" Nov 03 16:45:03.946874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.946856 5946 generic.go:356] "Generic (PLEG): container finished" podID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerID="eda0dfde12e72dd39d63cee301ad6acadcf07c40c1d8670408ce16f857812b49" exitCode=60 Nov 03 16:45:03.946980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.946879 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerDied","Data":"eda0dfde12e72dd39d63cee301ad6acadcf07c40c1d8670408ce16f857812b49"} Nov 03 16:45:03.947134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.947118 5946 scope.go:122] "RemoveContainer" containerID="eda0dfde12e72dd39d63cee301ad6acadcf07c40c1d8670408ce16f857812b49" Nov 03 16:45:03.949324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.949303 5946 generic.go:356] "Generic (PLEG): container finished" podID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerID="322e40faf4b7b8c7b3523f4ef4fed0c6dfdf330e8952ea330959e46fbadf7978" exitCode=60 Nov 03 16:45:03.949431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.949406 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerDied","Data":"322e40faf4b7b8c7b3523f4ef4fed0c6dfdf330e8952ea330959e46fbadf7978"} Nov 03 16:45:03.949551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.949536 5946 scope.go:122] "RemoveContainer" containerID="322e40faf4b7b8c7b3523f4ef4fed0c6dfdf330e8952ea330959e46fbadf7978" Nov 03 16:45:03.950970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.950952 5946 generic.go:356] "Generic (PLEG): container finished" podID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerID="525f807c3f6ab4af37333927ca6e589f9fa2425b0ced0cd3b0e68864bcb81e16" exitCode=60 Nov 03 16:45:03.951063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.951025 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" event={"ID":"4cd2a9eb-4be2-4031-822d-b49a4670b679","Type":"ContainerDied","Data":"525f807c3f6ab4af37333927ca6e589f9fa2425b0ced0cd3b0e68864bcb81e16"} Nov 03 16:45:03.951253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.951234 5946 scope.go:122] "RemoveContainer" containerID="525f807c3f6ab4af37333927ca6e589f9fa2425b0ced0cd3b0e68864bcb81e16" Nov 03 16:45:03.952771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.952749 5946 generic.go:356] "Generic (PLEG): container finished" podID="3de379cd-d26a-4e49-be70-f220543c177a" containerID="db9b97fa4717c731388971ed63581a75990805a2640b2a64d7569cb2ad81ea84" exitCode=60 Nov 03 16:45:03.952882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.952839 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerDied","Data":"db9b97fa4717c731388971ed63581a75990805a2640b2a64d7569cb2ad81ea84"} Nov 03 16:45:03.953091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.953070 5946 scope.go:122] "RemoveContainer" containerID="db9b97fa4717c731388971ed63581a75990805a2640b2a64d7569cb2ad81ea84" Nov 03 16:45:03.954642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.954621 5946 generic.go:356] "Generic (PLEG): container finished" podID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerID="a68244fb013d6af5901013906a7f46d642cce366c8611abb3d3ca5905af19dfb" exitCode=60 Nov 03 16:45:03.954737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.954705 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" event={"ID":"7101b7e0-dfa3-4b10-b60b-e515c861f40e","Type":"ContainerDied","Data":"a68244fb013d6af5901013906a7f46d642cce366c8611abb3d3ca5905af19dfb"} Nov 03 16:45:03.954901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.954864 5946 scope.go:122] "RemoveContainer" containerID="a68244fb013d6af5901013906a7f46d642cce366c8611abb3d3ca5905af19dfb" Nov 03 16:45:03.956352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.956334 5946 generic.go:356] "Generic (PLEG): container finished" podID="65d096c2-13d5-417c-a2d8-498581ab250f" containerID="816152a61dddc61e20fe4df62d40770d4a017dbd689989e6c8116cb78c83bc5d" exitCode=60 Nov 03 16:45:03.956450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.956397 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerDied","Data":"816152a61dddc61e20fe4df62d40770d4a017dbd689989e6c8116cb78c83bc5d"} Nov 03 16:45:03.956657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.956639 5946 scope.go:122] "RemoveContainer" containerID="816152a61dddc61e20fe4df62d40770d4a017dbd689989e6c8116cb78c83bc5d" Nov 03 16:45:03.962595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.962115 5946 generic.go:356] "Generic (PLEG): container finished" podID="ba75d816-f403-4721-acc8-60bddc30be52" containerID="6c566db842b5bd2c0afe8315d9702cdd83bc225d7591eefee79ae569644e5ecc" exitCode=60 Nov 03 16:45:03.962595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.962165 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" event={"ID":"ba75d816-f403-4721-acc8-60bddc30be52","Type":"ContainerDied","Data":"6c566db842b5bd2c0afe8315d9702cdd83bc225d7591eefee79ae569644e5ecc"} Nov 03 16:45:03.962595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.962471 5946 scope.go:122] "RemoveContainer" containerID="6c566db842b5bd2c0afe8315d9702cdd83bc225d7591eefee79ae569644e5ecc" Nov 03 16:45:03.983423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:03.983378 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" podStartSLOduration=3.983364783 podStartE2EDuration="3.983364783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.982797723 +0000 UTC m=+1974.103850547" watchObservedRunningTime="2026-11-03 16:45:03.983364783 +0000 UTC m=+1974.104417597" Nov 03 16:45:04.105300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.105173 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" podStartSLOduration=4.105159045 podStartE2EDuration="4.105159045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:04.101460678 +0000 UTC m=+1974.222513501" watchObservedRunningTime="2026-11-03 16:45:04.105159045 +0000 UTC m=+1974.226211865" Nov 03 16:45:04.225668 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.225623 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" podStartSLOduration=4.225608896 podStartE2EDuration="4.225608896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:04.223141704 +0000 UTC m=+1974.344194528" watchObservedRunningTime="2026-11-03 16:45:04.225608896 +0000 UTC m=+1974.346661722" Nov 03 16:45:04.346008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.345958 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" podStartSLOduration=4.345943533 podStartE2EDuration="4.345943533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:04.343913752 +0000 UTC m=+1974.464966575" watchObservedRunningTime="2026-11-03 16:45:04.345943533 +0000 UTC m=+1974.466996353" Nov 03 16:45:04.471259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.471165 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" podStartSLOduration=4.471146903 podStartE2EDuration="4.471146903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:04.466585189 +0000 UTC m=+1974.587638013" watchObservedRunningTime="2026-11-03 16:45:04.471146903 +0000 UTC m=+1974.592199725" Nov 03 16:45:04.590384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.590328 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" podStartSLOduration=4.590313126 podStartE2EDuration="4.590313126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:04.590011865 +0000 UTC m=+1974.711064689" watchObservedRunningTime="2026-11-03 16:45:04.590313126 +0000 UTC m=+1974.711365962" Nov 03 16:45:04.974167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.974079 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" event={"ID":"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5","Type":"ContainerStarted","Data":"5d122fced98a8cf32763caab514270370c7485e91da628a4c91e974a841a5e5e"} Nov 03 16:45:04.976293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.976264 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" event={"ID":"70db9170-6a1c-4b21-b509-67d85e1cee91","Type":"ContainerStarted","Data":"2fd02cba6a84683cebcb803ccd121029f21e88e4b05ee560a0a432afff75775a"} Nov 03 16:45:04.978231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.978186 5946 generic.go:356] "Generic (PLEG): container finished" podID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerID="74b0872b83425fabdd2759f325a1fba74d590199b4ba2bc0fde07580e1675a98" exitCode=60 Nov 03 16:45:04.978353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.978234 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" event={"ID":"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7","Type":"ContainerDied","Data":"74b0872b83425fabdd2759f325a1fba74d590199b4ba2bc0fde07580e1675a98"} Nov 03 16:45:04.978353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.978288 5946 scope.go:122] "RemoveContainer" containerID="e2c791d48c2716fdd7ae4aec3c4cbc52ec235d4e3a9a6b235df76245016bf3bd" Nov 03 16:45:04.978697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.978678 5946 scope.go:122] "RemoveContainer" containerID="74b0872b83425fabdd2759f325a1fba74d590199b4ba2bc0fde07580e1675a98" Nov 03 16:45:04.979031 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:04.979006 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng_odh-ai-gateway-infra(ce39a13e-32a3-4ab3-88bc-118b76cc2ea7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" Nov 03 16:45:04.980369 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.980350 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerStarted","Data":"8deee6754777e4baa5224902fd09c1c2b969037bca45e334969127ad2ba1e658"} Nov 03 16:45:04.982272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.982240 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerStarted","Data":"3662126fc8ff5253363cc14a2fce3b23e19b42e36f94c166934d1b80404d56b0"} Nov 03 16:45:04.984276 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.984236 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerStarted","Data":"5fc169de0a5c7e844599b004c1c16e87edf4b592dbf1ced0b8dddcb5f13aff8f"} Nov 03 16:45:04.986361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.986338 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerStarted","Data":"c08b696b8b2fd0662d83f5531224d6cadf2f9ce724531c876472a150c881202f"} Nov 03 16:45:04.988262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.988235 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" event={"ID":"4cd2a9eb-4be2-4031-822d-b49a4670b679","Type":"ContainerStarted","Data":"f9941bdd0dbc85e4c9892f8a0377fd1f2ebea610421c920c5ace6d2189fb278b"} Nov 03 16:45:04.990067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.990041 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerStarted","Data":"e497cb35e1f9b79a05c4d8e4d54c0175c52661fcf9387711943bb78fc68f5ba6"} Nov 03 16:45:04.991911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.991890 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" event={"ID":"7101b7e0-dfa3-4b10-b60b-e515c861f40e","Type":"ContainerStarted","Data":"900bec8396b32e5c8c45f3da7f2fa9c0e408dcb664de8748cb656662fea76a3e"} Nov 03 16:45:04.993722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.993701 5946 generic.go:356] "Generic (PLEG): container finished" podID="e82edb20-138b-4058-8303-e6b1655bc37e" containerID="a88c1d9d537599a2dc2ece9cfbae147285fe43173b67d580e576da4e7e649526" exitCode=60 Nov 03 16:45:04.993821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.993766 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" event={"ID":"e82edb20-138b-4058-8303-e6b1655bc37e","Type":"ContainerDied","Data":"a88c1d9d537599a2dc2ece9cfbae147285fe43173b67d580e576da4e7e649526"} Nov 03 16:45:04.994012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.993973 5946 scope.go:122] "RemoveContainer" containerID="a88c1d9d537599a2dc2ece9cfbae147285fe43173b67d580e576da4e7e649526" Nov 03 16:45:04.994078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.994061 5946 scope.go:122] "RemoveContainer" containerID="c14585da915cba467e7f6a657e2839b46d34bceb4080a89423300ce30519a2a6" Nov 03 16:45:04.994238 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:04.994198 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9_odh-ai-gateway-infra(e82edb20-138b-4058-8303-e6b1655bc37e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" Nov 03 16:45:04.995721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.995702 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerStarted","Data":"b2d7a811a62102ec57a3e23c5dd76441577e272934e1482f0e83a49af36e7686"} Nov 03 16:45:04.997672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.997644 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" event={"ID":"ba75d816-f403-4721-acc8-60bddc30be52","Type":"ContainerStarted","Data":"76895f50bb507da3d4f6410c13ff10fbbd43a53fe5e8d001685e673b2689340f"} Nov 03 16:45:04.999553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:04.999535 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerStarted","Data":"c559a7e69c51ee9abd587d00c148260f3d777c4a33621a07068a004b27b1de1d"} Nov 03 16:45:06.009267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.009232 5946 generic.go:356] "Generic (PLEG): container finished" podID="3de379cd-d26a-4e49-be70-f220543c177a" containerID="e497cb35e1f9b79a05c4d8e4d54c0175c52661fcf9387711943bb78fc68f5ba6" exitCode=60 Nov 03 16:45:06.009717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.009322 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerDied","Data":"e497cb35e1f9b79a05c4d8e4d54c0175c52661fcf9387711943bb78fc68f5ba6"} Nov 03 16:45:06.009717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.009387 5946 scope.go:122] "RemoveContainer" containerID="db9b97fa4717c731388971ed63581a75990805a2640b2a64d7569cb2ad81ea84" Nov 03 16:45:06.009717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.009657 5946 scope.go:122] "RemoveContainer" containerID="e497cb35e1f9b79a05c4d8e4d54c0175c52661fcf9387711943bb78fc68f5ba6" Nov 03 16:45:06.010034 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.010015 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz_odh-ai-gateway-infra(3de379cd-d26a-4e49-be70-f220543c177a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" podUID="3de379cd-d26a-4e49-be70-f220543c177a" Nov 03 16:45:06.011510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.011486 5946 generic.go:356] "Generic (PLEG): container finished" podID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerID="900bec8396b32e5c8c45f3da7f2fa9c0e408dcb664de8748cb656662fea76a3e" exitCode=60 Nov 03 16:45:06.011632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.011570 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" event={"ID":"7101b7e0-dfa3-4b10-b60b-e515c861f40e","Type":"ContainerDied","Data":"900bec8396b32e5c8c45f3da7f2fa9c0e408dcb664de8748cb656662fea76a3e"} Nov 03 16:45:06.011802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.011787 5946 scope.go:122] "RemoveContainer" containerID="900bec8396b32e5c8c45f3da7f2fa9c0e408dcb664de8748cb656662fea76a3e" Nov 03 16:45:06.012108 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.012089 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-gpl78_odh-ai-gateway-infra(7101b7e0-dfa3-4b10-b60b-e515c861f40e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" Nov 03 16:45:06.013419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.013403 5946 scope.go:122] "RemoveContainer" containerID="a88c1d9d537599a2dc2ece9cfbae147285fe43173b67d580e576da4e7e649526" Nov 03 16:45:06.013712 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.013693 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9_odh-ai-gateway-infra(e82edb20-138b-4058-8303-e6b1655bc37e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" Nov 03 16:45:06.015158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.015135 5946 generic.go:356] "Generic (PLEG): container finished" podID="65d096c2-13d5-417c-a2d8-498581ab250f" containerID="b2d7a811a62102ec57a3e23c5dd76441577e272934e1482f0e83a49af36e7686" exitCode=60 Nov 03 16:45:06.015295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.015226 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerDied","Data":"b2d7a811a62102ec57a3e23c5dd76441577e272934e1482f0e83a49af36e7686"} Nov 03 16:45:06.015447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.015430 5946 scope.go:122] "RemoveContainer" containerID="b2d7a811a62102ec57a3e23c5dd76441577e272934e1482f0e83a49af36e7686" Nov 03 16:45:06.015700 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.015679 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f_odh-ai-gateway-infra(65d096c2-13d5-417c-a2d8-498581ab250f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" Nov 03 16:45:06.017237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.017199 5946 generic.go:356] "Generic (PLEG): container finished" podID="ba75d816-f403-4721-acc8-60bddc30be52" containerID="76895f50bb507da3d4f6410c13ff10fbbd43a53fe5e8d001685e673b2689340f" exitCode=60 Nov 03 16:45:06.017324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.017244 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" event={"ID":"ba75d816-f403-4721-acc8-60bddc30be52","Type":"ContainerDied","Data":"76895f50bb507da3d4f6410c13ff10fbbd43a53fe5e8d001685e673b2689340f"} Nov 03 16:45:06.017674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.017653 5946 scope.go:122] "RemoveContainer" containerID="76895f50bb507da3d4f6410c13ff10fbbd43a53fe5e8d001685e673b2689340f" Nov 03 16:45:06.018003 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.017966 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7_odh-ai-gateway-infra(ba75d816-f403-4721-acc8-60bddc30be52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" podUID="ba75d816-f403-4721-acc8-60bddc30be52" Nov 03 16:45:06.019637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.019617 5946 generic.go:356] "Generic (PLEG): container finished" podID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerID="c559a7e69c51ee9abd587d00c148260f3d777c4a33621a07068a004b27b1de1d" exitCode=60 Nov 03 16:45:06.019736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.019640 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerDied","Data":"c559a7e69c51ee9abd587d00c148260f3d777c4a33621a07068a004b27b1de1d"} Nov 03 16:45:06.019903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.019889 5946 scope.go:122] "RemoveContainer" containerID="c559a7e69c51ee9abd587d00c148260f3d777c4a33621a07068a004b27b1de1d" Nov 03 16:45:06.020147 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.020123 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d_odh-ai-gateway-infra(ccbfe05c-7d59-4e1a-81ed-f480be5892d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" Nov 03 16:45:06.021660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.021639 5946 generic.go:356] "Generic (PLEG): container finished" podID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerID="5d122fced98a8cf32763caab514270370c7485e91da628a4c91e974a841a5e5e" exitCode=60 Nov 03 16:45:06.021758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.021702 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" event={"ID":"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5","Type":"ContainerDied","Data":"5d122fced98a8cf32763caab514270370c7485e91da628a4c91e974a841a5e5e"} Nov 03 16:45:06.021946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.021934 5946 scope.go:122] "RemoveContainer" containerID="5d122fced98a8cf32763caab514270370c7485e91da628a4c91e974a841a5e5e" Nov 03 16:45:06.022245 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.022222 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr_odh-ai-gateway-infra(bd3a99a6-3244-414d-aaba-1ac35a4ec7a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" Nov 03 16:45:06.024193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.024094 5946 generic.go:356] "Generic (PLEG): container finished" podID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerID="2fd02cba6a84683cebcb803ccd121029f21e88e4b05ee560a0a432afff75775a" exitCode=60 Nov 03 16:45:06.024193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.024170 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" event={"ID":"70db9170-6a1c-4b21-b509-67d85e1cee91","Type":"ContainerDied","Data":"2fd02cba6a84683cebcb803ccd121029f21e88e4b05ee560a0a432afff75775a"} Nov 03 16:45:06.024519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.024496 5946 scope.go:122] "RemoveContainer" containerID="2fd02cba6a84683cebcb803ccd121029f21e88e4b05ee560a0a432afff75775a" Nov 03 16:45:06.024813 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.024792 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh_odh-ai-gateway-infra(70db9170-6a1c-4b21-b509-67d85e1cee91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" Nov 03 16:45:06.030606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.028494 5946 scope.go:122] "RemoveContainer" containerID="a68244fb013d6af5901013906a7f46d642cce366c8611abb3d3ca5905af19dfb" Nov 03 16:45:06.034491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.034463 5946 generic.go:356] "Generic (PLEG): container finished" podID="b5836760-c55b-46d4-9050-f33870943c0b" containerID="8deee6754777e4baa5224902fd09c1c2b969037bca45e334969127ad2ba1e658" exitCode=60 Nov 03 16:45:06.034613 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.034542 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerDied","Data":"8deee6754777e4baa5224902fd09c1c2b969037bca45e334969127ad2ba1e658"} Nov 03 16:45:06.034823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.034805 5946 scope.go:122] "RemoveContainer" containerID="8deee6754777e4baa5224902fd09c1c2b969037bca45e334969127ad2ba1e658" Nov 03 16:45:06.035232 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.035183 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb_odh-ai-gateway-infra(b5836760-c55b-46d4-9050-f33870943c0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" podUID="b5836760-c55b-46d4-9050-f33870943c0b" Nov 03 16:45:06.036445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.036421 5946 generic.go:356] "Generic (PLEG): container finished" podID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerID="3662126fc8ff5253363cc14a2fce3b23e19b42e36f94c166934d1b80404d56b0" exitCode=60 Nov 03 16:45:06.036579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.036558 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerDied","Data":"3662126fc8ff5253363cc14a2fce3b23e19b42e36f94c166934d1b80404d56b0"} Nov 03 16:45:06.036861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.036844 5946 scope.go:122] "RemoveContainer" containerID="3662126fc8ff5253363cc14a2fce3b23e19b42e36f94c166934d1b80404d56b0" Nov 03 16:45:06.037157 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.037137 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw_odh-ai-gateway-infra(9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" Nov 03 16:45:06.038554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.038535 5946 generic.go:356] "Generic (PLEG): container finished" podID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerID="5fc169de0a5c7e844599b004c1c16e87edf4b592dbf1ced0b8dddcb5f13aff8f" exitCode=60 Nov 03 16:45:06.038652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.038615 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerDied","Data":"5fc169de0a5c7e844599b004c1c16e87edf4b592dbf1ced0b8dddcb5f13aff8f"} Nov 03 16:45:06.038886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.038869 5946 scope.go:122] "RemoveContainer" containerID="5fc169de0a5c7e844599b004c1c16e87edf4b592dbf1ced0b8dddcb5f13aff8f" Nov 03 16:45:06.039219 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.039181 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r_odh-ai-gateway-infra(eefd3d1f-d7e6-4bef-b033-508b910c0a2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" Nov 03 16:45:06.040602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.040583 5946 generic.go:356] "Generic (PLEG): container finished" podID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerID="c08b696b8b2fd0662d83f5531224d6cadf2f9ce724531c876472a150c881202f" exitCode=60 Nov 03 16:45:06.040737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.040720 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerDied","Data":"c08b696b8b2fd0662d83f5531224d6cadf2f9ce724531c876472a150c881202f"} Nov 03 16:45:06.041113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.041090 5946 scope.go:122] "RemoveContainer" containerID="c08b696b8b2fd0662d83f5531224d6cadf2f9ce724531c876472a150c881202f" Nov 03 16:45:06.041386 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.041364 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84_odh-ai-gateway-infra(ce2f4ad1-baad-41c9-a3e4-923dcae51c41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" Nov 03 16:45:06.043000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.042984 5946 generic.go:356] "Generic (PLEG): container finished" podID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerID="f9941bdd0dbc85e4c9892f8a0377fd1f2ebea610421c920c5ace6d2189fb278b" exitCode=60 Nov 03 16:45:06.043088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.043018 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" event={"ID":"4cd2a9eb-4be2-4031-822d-b49a4670b679","Type":"ContainerDied","Data":"f9941bdd0dbc85e4c9892f8a0377fd1f2ebea610421c920c5ace6d2189fb278b"} Nov 03 16:45:06.043258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.043244 5946 scope.go:122] "RemoveContainer" containerID="f9941bdd0dbc85e4c9892f8a0377fd1f2ebea610421c920c5ace6d2189fb278b" Nov 03 16:45:06.043472 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:06.043455 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf_odh-ai-gateway-infra(4cd2a9eb-4be2-4031-822d-b49a4670b679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" Nov 03 16:45:06.047063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.047030 5946 scope.go:122] "RemoveContainer" containerID="816152a61dddc61e20fe4df62d40770d4a017dbd689989e6c8116cb78c83bc5d" Nov 03 16:45:06.061050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.061028 5946 scope.go:122] "RemoveContainer" containerID="6c566db842b5bd2c0afe8315d9702cdd83bc225d7591eefee79ae569644e5ecc" Nov 03 16:45:06.075260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.075238 5946 scope.go:122] "RemoveContainer" containerID="07cd07e79d04987d7099b0674df3b99c0289b5ad0eb6f3369e4121331b1ab491" Nov 03 16:45:06.088965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.088938 5946 scope.go:122] "RemoveContainer" containerID="71839896a60a83a47393e5a80f8f8b2bed136cd50ec1f5a9dfdc8e7cc4ee503f" Nov 03 16:45:06.102961 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.102937 5946 scope.go:122] "RemoveContainer" containerID="ed5df2f6b1782a19180fb69d7bd4e71716c31d1c414e50ec6e75763477efc4dd" Nov 03 16:45:06.116479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.116460 5946 scope.go:122] "RemoveContainer" containerID="b69ef8a059d620dcd970e0dffd01039fb5c6db27a664b8d1870e24c3b5c55018" Nov 03 16:45:06.130065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.130043 5946 scope.go:122] "RemoveContainer" containerID="436f543620d0fd7f00902e37865badd21b254a51df1d27a147c4536a4f6de5b6" Nov 03 16:45:06.144545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.144516 5946 scope.go:122] "RemoveContainer" containerID="eda0dfde12e72dd39d63cee301ad6acadcf07c40c1d8670408ce16f857812b49" Nov 03 16:45:06.170037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.170001 5946 scope.go:122] "RemoveContainer" containerID="322e40faf4b7b8c7b3523f4ef4fed0c6dfdf330e8952ea330959e46fbadf7978" Nov 03 16:45:06.184506 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:06.184481 5946 scope.go:122] "RemoveContainer" containerID="525f807c3f6ab4af37333927ca6e589f9fa2425b0ced0cd3b0e68864bcb81e16" Nov 03 16:45:07.053399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.053362 5946 scope.go:122] "RemoveContainer" containerID="c559a7e69c51ee9abd587d00c148260f3d777c4a33621a07068a004b27b1de1d" Nov 03 16:45:07.053887 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.053677 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d_odh-ai-gateway-infra(ccbfe05c-7d59-4e1a-81ed-f480be5892d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" Nov 03 16:45:07.054992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.054962 5946 scope.go:122] "RemoveContainer" containerID="5d122fced98a8cf32763caab514270370c7485e91da628a4c91e974a841a5e5e" Nov 03 16:45:07.055246 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.055221 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr_odh-ai-gateway-infra(bd3a99a6-3244-414d-aaba-1ac35a4ec7a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" Nov 03 16:45:07.056554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.056538 5946 scope.go:122] "RemoveContainer" containerID="2fd02cba6a84683cebcb803ccd121029f21e88e4b05ee560a0a432afff75775a" Nov 03 16:45:07.056758 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.056736 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh_odh-ai-gateway-infra(70db9170-6a1c-4b21-b509-67d85e1cee91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" Nov 03 16:45:07.058110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.058092 5946 scope.go:122] "RemoveContainer" containerID="8deee6754777e4baa5224902fd09c1c2b969037bca45e334969127ad2ba1e658" Nov 03 16:45:07.058364 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.058340 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb_odh-ai-gateway-infra(b5836760-c55b-46d4-9050-f33870943c0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" podUID="b5836760-c55b-46d4-9050-f33870943c0b" Nov 03 16:45:07.059537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.059518 5946 scope.go:122] "RemoveContainer" containerID="3662126fc8ff5253363cc14a2fce3b23e19b42e36f94c166934d1b80404d56b0" Nov 03 16:45:07.059727 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.059705 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw_odh-ai-gateway-infra(9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" Nov 03 16:45:07.061047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.061030 5946 scope.go:122] "RemoveContainer" containerID="5fc169de0a5c7e844599b004c1c16e87edf4b592dbf1ced0b8dddcb5f13aff8f" Nov 03 16:45:07.061293 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.061272 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r_odh-ai-gateway-infra(eefd3d1f-d7e6-4bef-b033-508b910c0a2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" Nov 03 16:45:07.062520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.062506 5946 scope.go:122] "RemoveContainer" containerID="c08b696b8b2fd0662d83f5531224d6cadf2f9ce724531c876472a150c881202f" Nov 03 16:45:07.062717 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.062698 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84_odh-ai-gateway-infra(ce2f4ad1-baad-41c9-a3e4-923dcae51c41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" Nov 03 16:45:07.064064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.064047 5946 scope.go:122] "RemoveContainer" containerID="f9941bdd0dbc85e4c9892f8a0377fd1f2ebea610421c920c5ace6d2189fb278b" Nov 03 16:45:07.064311 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.064285 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf_odh-ai-gateway-infra(4cd2a9eb-4be2-4031-822d-b49a4670b679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" Nov 03 16:45:07.065885 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.065866 5946 scope.go:122] "RemoveContainer" containerID="e497cb35e1f9b79a05c4d8e4d54c0175c52661fcf9387711943bb78fc68f5ba6" Nov 03 16:45:07.066074 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.066055 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz_odh-ai-gateway-infra(3de379cd-d26a-4e49-be70-f220543c177a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" podUID="3de379cd-d26a-4e49-be70-f220543c177a" Nov 03 16:45:07.067440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.067424 5946 scope.go:122] "RemoveContainer" containerID="900bec8396b32e5c8c45f3da7f2fa9c0e408dcb664de8748cb656662fea76a3e" Nov 03 16:45:07.067650 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.067626 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-gpl78_odh-ai-gateway-infra(7101b7e0-dfa3-4b10-b60b-e515c861f40e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" Nov 03 16:45:07.069008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.068994 5946 scope.go:122] "RemoveContainer" containerID="b2d7a811a62102ec57a3e23c5dd76441577e272934e1482f0e83a49af36e7686" Nov 03 16:45:07.069190 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.069171 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f_odh-ai-gateway-infra(65d096c2-13d5-417c-a2d8-498581ab250f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" Nov 03 16:45:07.070588 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.070572 5946 scope.go:122] "RemoveContainer" containerID="76895f50bb507da3d4f6410c13ff10fbbd43a53fe5e8d001685e673b2689340f" Nov 03 16:45:07.070791 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:07.070771 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7_odh-ai-gateway-infra(ba75d816-f403-4721-acc8-60bddc30be52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" podUID="ba75d816-f403-4721-acc8-60bddc30be52" Nov 03 16:45:07.558897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:07.558867 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:09.329944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:09.329912 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:11.619801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:11.619766 5946 scope.go:122] "RemoveContainer" containerID="d432572032767c454740f34cefd05e99063839d0b878e3ec1a82edc66e6f9d57" Nov 03 16:45:11.630335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:11.630313 5946 scope.go:122] "RemoveContainer" containerID="a2c610d9a71ceabfde905d3ebcd31c52b31c24dd35d4b4bee560bfab58ea4ab5" Nov 03 16:45:11.640459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:11.640436 5946 scope.go:122] "RemoveContainer" containerID="c1e0b3f1ba8e9ca5304c0719594173d3fc3681b2f4a4a140649ffd79c9049340" Nov 03 16:45:12.455607 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:12.455547 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:12.587117 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:12.587072 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:45:14.072557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:14.072518 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2"] Nov 03 16:45:14.122581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:14.122549 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f39b18f-65lc2"] Nov 03 16:45:14.475121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:14.475030 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:45:15.001830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:15.001798 5946 scope.go:122] "RemoveContainer" containerID="a88c1d9d537599a2dc2ece9cfbae147285fe43173b67d580e576da4e7e649526" Nov 03 16:45:16.006909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:16.006875 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e550cc-f09e-4c22-87db-dda7f442ebc6" path="/var/lib/kubelet/pods/a9e550cc-f09e-4c22-87db-dda7f442ebc6/volumes" Nov 03 16:45:16.007800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:16.007784 5946 scope.go:122] "RemoveContainer" containerID="c08b696b8b2fd0662d83f5531224d6cadf2f9ce724531c876472a150c881202f" Nov 03 16:45:16.007944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:16.007928 5946 scope.go:122] "RemoveContainer" containerID="c559a7e69c51ee9abd587d00c148260f3d777c4a33621a07068a004b27b1de1d" Nov 03 16:45:16.152081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:16.152049 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" event={"ID":"e82edb20-138b-4058-8303-e6b1655bc37e","Type":"ContainerStarted","Data":"3b81d6c086626c25a1807d857c417a7e8160bdff211a4743f7cc77daff56e9af"} Nov 03 16:45:16.997932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:16.997908 5946 scope.go:122] "RemoveContainer" containerID="f9941bdd0dbc85e4c9892f8a0377fd1f2ebea610421c920c5ace6d2189fb278b" Nov 03 16:45:16.998097 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:16.998048 5946 scope.go:122] "RemoveContainer" containerID="5d122fced98a8cf32763caab514270370c7485e91da628a4c91e974a841a5e5e" Nov 03 16:45:17.162125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.162083 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerStarted","Data":"2da01c0698e7cbec36486a9dcc526a7053fb8047b702f7263125954379f8b88e"} Nov 03 16:45:17.164000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.163973 5946 generic.go:356] "Generic (PLEG): container finished" podID="e82edb20-138b-4058-8303-e6b1655bc37e" containerID="3b81d6c086626c25a1807d857c417a7e8160bdff211a4743f7cc77daff56e9af" exitCode=60 Nov 03 16:45:17.164130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.164043 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" event={"ID":"e82edb20-138b-4058-8303-e6b1655bc37e","Type":"ContainerDied","Data":"3b81d6c086626c25a1807d857c417a7e8160bdff211a4743f7cc77daff56e9af"} Nov 03 16:45:17.164130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.164089 5946 scope.go:122] "RemoveContainer" containerID="a88c1d9d537599a2dc2ece9cfbae147285fe43173b67d580e576da4e7e649526" Nov 03 16:45:17.164471 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.164455 5946 scope.go:122] "RemoveContainer" containerID="3b81d6c086626c25a1807d857c417a7e8160bdff211a4743f7cc77daff56e9af" Nov 03 16:45:17.164800 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:17.164776 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9_odh-ai-gateway-infra(e82edb20-138b-4058-8303-e6b1655bc37e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" Nov 03 16:45:17.166311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.166289 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerStarted","Data":"58d98e23434ffd6234fe5026dc32ef8045bf47fe850c1cb22e40ab8fd646e110"} Nov 03 16:45:17.289856 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.289826 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9"] Nov 03 16:45:17.398907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.398857 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:17.997551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.997519 5946 scope.go:122] "RemoveContainer" containerID="5fc169de0a5c7e844599b004c1c16e87edf4b592dbf1ced0b8dddcb5f13aff8f" Nov 03 16:45:17.997758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.997660 5946 scope.go:122] "RemoveContainer" containerID="3662126fc8ff5253363cc14a2fce3b23e19b42e36f94c166934d1b80404d56b0" Nov 03 16:45:17.997916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.997898 5946 scope.go:122] "RemoveContainer" containerID="8deee6754777e4baa5224902fd09c1c2b969037bca45e334969127ad2ba1e658" Nov 03 16:45:17.998001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.997982 5946 scope.go:122] "RemoveContainer" containerID="e497cb35e1f9b79a05c4d8e4d54c0175c52661fcf9387711943bb78fc68f5ba6" Nov 03 16:45:17.998116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.998098 5946 scope.go:122] "RemoveContainer" containerID="b2d7a811a62102ec57a3e23c5dd76441577e272934e1482f0e83a49af36e7686" Nov 03 16:45:17.998199 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:17.998182 5946 scope.go:122] "RemoveContainer" containerID="74b0872b83425fabdd2759f325a1fba74d590199b4ba2bc0fde07580e1675a98" Nov 03 16:45:18.180619 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.180581 5946 generic.go:356] "Generic (PLEG): container finished" podID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerID="2da01c0698e7cbec36486a9dcc526a7053fb8047b702f7263125954379f8b88e" exitCode=60 Nov 03 16:45:18.180994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.180692 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerDied","Data":"2da01c0698e7cbec36486a9dcc526a7053fb8047b702f7263125954379f8b88e"} Nov 03 16:45:18.180994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.180749 5946 scope.go:122] "RemoveContainer" containerID="c08b696b8b2fd0662d83f5531224d6cadf2f9ce724531c876472a150c881202f" Nov 03 16:45:18.181323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.181304 5946 scope.go:122] "RemoveContainer" containerID="2da01c0698e7cbec36486a9dcc526a7053fb8047b702f7263125954379f8b88e" Nov 03 16:45:18.181755 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:18.181729 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84_odh-ai-gateway-infra(ce2f4ad1-baad-41c9-a3e4-923dcae51c41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" Nov 03 16:45:18.184634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.184609 5946 generic.go:356] "Generic (PLEG): container finished" podID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerID="2bf5234a730d7c64cb841e8aebcd15ec62e64a828494d68b688dd0cc36062ad5" exitCode=60 Nov 03 16:45:18.184760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.184677 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" event={"ID":"4cd2a9eb-4be2-4031-822d-b49a4670b679","Type":"ContainerDied","Data":"2bf5234a730d7c64cb841e8aebcd15ec62e64a828494d68b688dd0cc36062ad5"} Nov 03 16:45:18.185174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.185155 5946 scope.go:122] "RemoveContainer" containerID="2bf5234a730d7c64cb841e8aebcd15ec62e64a828494d68b688dd0cc36062ad5" Nov 03 16:45:18.186006 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:18.185522 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf_odh-ai-gateway-infra(4cd2a9eb-4be2-4031-822d-b49a4670b679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" Nov 03 16:45:18.191806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.191787 5946 generic.go:356] "Generic (PLEG): container finished" podID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerID="58d98e23434ffd6234fe5026dc32ef8045bf47fe850c1cb22e40ab8fd646e110" exitCode=60 Nov 03 16:45:18.191892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.191861 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerDied","Data":"58d98e23434ffd6234fe5026dc32ef8045bf47fe850c1cb22e40ab8fd646e110"} Nov 03 16:45:18.192418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.192400 5946 scope.go:122] "RemoveContainer" containerID="58d98e23434ffd6234fe5026dc32ef8045bf47fe850c1cb22e40ab8fd646e110" Nov 03 16:45:18.192760 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:18.192736 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d_odh-ai-gateway-infra(ccbfe05c-7d59-4e1a-81ed-f480be5892d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" Nov 03 16:45:18.194158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.194139 5946 generic.go:356] "Generic (PLEG): container finished" podID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerID="aba0f248bdd312f20f2e841b0f2fb32387989bb6b5491f31a5968862e3b93f9a" exitCode=60 Nov 03 16:45:18.194273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.194171 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" event={"ID":"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5","Type":"ContainerDied","Data":"aba0f248bdd312f20f2e841b0f2fb32387989bb6b5491f31a5968862e3b93f9a"} Nov 03 16:45:18.194571 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.194557 5946 scope.go:122] "RemoveContainer" containerID="aba0f248bdd312f20f2e841b0f2fb32387989bb6b5491f31a5968862e3b93f9a" Nov 03 16:45:18.194826 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:18.194810 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr_odh-ai-gateway-infra(bd3a99a6-3244-414d-aaba-1ac35a4ec7a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" Nov 03 16:45:18.267262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.266307 5946 scope.go:122] "RemoveContainer" containerID="f9941bdd0dbc85e4c9892f8a0377fd1f2ebea610421c920c5ace6d2189fb278b" Nov 03 16:45:18.300440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.300371 5946 scope.go:122] "RemoveContainer" containerID="c559a7e69c51ee9abd587d00c148260f3d777c4a33621a07068a004b27b1de1d" Nov 03 16:45:18.354592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.354565 5946 scope.go:122] "RemoveContainer" containerID="5d122fced98a8cf32763caab514270370c7485e91da628a4c91e974a841a5e5e" Nov 03 16:45:18.356560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.356539 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84"] Nov 03 16:45:18.443259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.443229 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d"] Nov 03 16:45:18.489921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.489893 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:18.646483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.646442 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82edb20-138b-4058-8303-e6b1655bc37e-kube-api-access-566f8\" (UniqueName: \"kubernetes.io/projected/e82edb20-138b-4058-8303-e6b1655bc37e-kube-api-access-566f8\") pod \"e82edb20-138b-4058-8303-e6b1655bc37e\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " Nov 03 16:45:18.646739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.646708 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82edb20-138b-4058-8303-e6b1655bc37e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82edb20-138b-4058-8303-e6b1655bc37e-maas-api-service-ca\") pod \"e82edb20-138b-4058-8303-e6b1655bc37e\" (UID: \"e82edb20-138b-4058-8303-e6b1655bc37e\") " Nov 03 16:45:18.649548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.649512 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82edb20-138b-4058-8303-e6b1655bc37e-maas-api-service-ca" pod "e82edb20-138b-4058-8303-e6b1655bc37e" (UID: "e82edb20-138b-4058-8303-e6b1655bc37e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:18.737457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.737218 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82edb20-138b-4058-8303-e6b1655bc37e-kube-api-access-566f8" pod "e82edb20-138b-4058-8303-e6b1655bc37e" (UID: "e82edb20-138b-4058-8303-e6b1655bc37e"). InnerVolumeSpecName "kube-api-access-566f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:18.755171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.755140 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566f8\" (UniqueName: \"kubernetes.io/projected/e82edb20-138b-4058-8303-e6b1655bc37e-kube-api-access-566f8\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:18.755323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.755180 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82edb20-138b-4058-8303-e6b1655bc37e-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:18.997278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:18.997244 5946 scope.go:122] "RemoveContainer" containerID="2fd02cba6a84683cebcb803ccd121029f21e88e4b05ee560a0a432afff75775a" Nov 03 16:45:19.203372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.203296 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" Nov 03 16:45:19.203784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.203295 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9" event={"ID":"e82edb20-138b-4058-8303-e6b1655bc37e","Type":"ContainerDied","Data":"305c3cd4d27cb855a34458463aae654c2b31905a61bd4673e07e6bba2a5921bb"} Nov 03 16:45:19.203784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.203448 5946 scope.go:122] "RemoveContainer" containerID="3b81d6c086626c25a1807d857c417a7e8160bdff211a4743f7cc77daff56e9af" Nov 03 16:45:19.205525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.205499 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerStarted","Data":"f5c55e03361c6c6eafa57ebf72ab3031196317a60bbbbabaf95be4f485b61ca8"} Nov 03 16:45:19.212746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.212722 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" event={"ID":"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7","Type":"ContainerStarted","Data":"230897a9b797396c55b7a8c15bf0bfdd336ac67dcc755d824353e2e6cdaf3155"} Nov 03 16:45:19.226546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.224324 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerStarted","Data":"15705c1b2178c0d19aaf712b2fa7ca83265a74730d05c438f17af7a18a1f6503"} Nov 03 16:45:19.255061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.254245 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerStarted","Data":"57cf9953d56d3eb2d2892c6fe1a2dc17bf9036cfccc1cf0df59c74f609514978"} Nov 03 16:45:19.259992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.259294 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerStarted","Data":"6c88dd5e4c08d92c3a28fb44f738e850e64eb351c7dbe05eb643c133f9340076"} Nov 03 16:45:19.268851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.268613 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerStarted","Data":"079547130a0e1f021c983400139640824bb0d584a9bb2fa47273ff2109947217"} Nov 03 16:45:19.391252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.391181 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9"] Nov 03 16:45:19.401699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.401663 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-438248-29895405-ksnk9"] Nov 03 16:45:19.499478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.499169 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr"] Nov 03 16:45:19.847589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.847303 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf"] Nov 03 16:45:19.852909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.852883 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:19.856616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.856594 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:19.860720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.860698 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:19.894631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.894469 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-kube-api-access-hpnlv\" (UniqueName: \"kubernetes.io/projected/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-kube-api-access-hpnlv\") pod \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " Nov 03 16:45:19.894631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.894587 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-kube-api-access-wmrf6\" (UniqueName: \"kubernetes.io/projected/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-kube-api-access-wmrf6\") pod \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " Nov 03 16:45:19.894810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.894647 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-maas-api-service-ca\") pod \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\" (UID: \"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5\") " Nov 03 16:45:19.894810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.894670 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-maas-api-service-ca\") pod \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " Nov 03 16:45:19.894810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.894700 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-maas-api-service-ca\") pod \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\" (UID: \"ce2f4ad1-baad-41c9-a3e4-923dcae51c41\") " Nov 03 16:45:19.894921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.894832 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-kube-api-access-g55ps\" (UniqueName: \"kubernetes.io/projected/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-kube-api-access-g55ps\") pod \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\" (UID: \"ccbfe05c-7d59-4e1a-81ed-f480be5892d0\") " Nov 03 16:45:19.895235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.895044 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-maas-api-service-ca" pod "bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" (UID: "bd3a99a6-3244-414d-aaba-1ac35a4ec7a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:19.895699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.895429 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:19.895699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.895533 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-maas-api-service-ca" pod "ce2f4ad1-baad-41c9-a3e4-923dcae51c41" (UID: "ce2f4ad1-baad-41c9-a3e4-923dcae51c41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:19.895699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.895651 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-maas-api-service-ca" pod "ccbfe05c-7d59-4e1a-81ed-f480be5892d0" (UID: "ccbfe05c-7d59-4e1a-81ed-f480be5892d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:19.897182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.897133 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-kube-api-access-wmrf6" pod "ce2f4ad1-baad-41c9-a3e4-923dcae51c41" (UID: "ce2f4ad1-baad-41c9-a3e4-923dcae51c41"). InnerVolumeSpecName "kube-api-access-wmrf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:19.897494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.897397 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-kube-api-access-hpnlv" pod "bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" (UID: "bd3a99a6-3244-414d-aaba-1ac35a4ec7a5"). InnerVolumeSpecName "kube-api-access-hpnlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:19.897558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.897496 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-kube-api-access-g55ps" pod "ccbfe05c-7d59-4e1a-81ed-f480be5892d0" (UID: "ccbfe05c-7d59-4e1a-81ed-f480be5892d0"). InnerVolumeSpecName "kube-api-access-g55ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:19.996843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.996812 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g55ps\" (UniqueName: \"kubernetes.io/projected/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-kube-api-access-g55ps\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:19.996843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.996842 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnlv\" (UniqueName: \"kubernetes.io/projected/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5-kube-api-access-hpnlv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:19.997092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.996852 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrf6\" (UniqueName: \"kubernetes.io/projected/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-kube-api-access-wmrf6\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:19.997092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.996863 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbfe05c-7d59-4e1a-81ed-f480be5892d0-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:19.997092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.996872 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f4ad1-baad-41c9-a3e4-923dcae51c41-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:19.997689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.997666 5946 scope.go:122] "RemoveContainer" containerID="900bec8396b32e5c8c45f3da7f2fa9c0e408dcb664de8748cb656662fea76a3e" Nov 03 16:45:19.998284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:19.998261 5946 scope.go:122] "RemoveContainer" containerID="76895f50bb507da3d4f6410c13ff10fbbd43a53fe5e8d001685e673b2689340f" Nov 03 16:45:20.051757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.051724 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" path="/var/lib/kubelet/pods/e82edb20-138b-4058-8303-e6b1655bc37e/volumes" Nov 03 16:45:20.056085 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.056052 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:20.097592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.097558 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd2a9eb-4be2-4031-822d-b49a4670b679-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a9eb-4be2-4031-822d-b49a4670b679-maas-api-service-ca\") pod \"4cd2a9eb-4be2-4031-822d-b49a4670b679\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " Nov 03 16:45:20.097705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.097689 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd2a9eb-4be2-4031-822d-b49a4670b679-kube-api-access-mw89r\" (UniqueName: \"kubernetes.io/projected/4cd2a9eb-4be2-4031-822d-b49a4670b679-kube-api-access-mw89r\") pod \"4cd2a9eb-4be2-4031-822d-b49a4670b679\" (UID: \"4cd2a9eb-4be2-4031-822d-b49a4670b679\") " Nov 03 16:45:20.097898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.097872 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd2a9eb-4be2-4031-822d-b49a4670b679-maas-api-service-ca" pod "4cd2a9eb-4be2-4031-822d-b49a4670b679" (UID: "4cd2a9eb-4be2-4031-822d-b49a4670b679"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:20.098478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.098443 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a9eb-4be2-4031-822d-b49a4670b679-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:20.099909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.099886 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd2a9eb-4be2-4031-822d-b49a4670b679-kube-api-access-mw89r" pod "4cd2a9eb-4be2-4031-822d-b49a4670b679" (UID: "4cd2a9eb-4be2-4031-822d-b49a4670b679"). InnerVolumeSpecName "kube-api-access-mw89r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:20.199968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.199936 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw89r\" (UniqueName: \"kubernetes.io/projected/4cd2a9eb-4be2-4031-822d-b49a4670b679-kube-api-access-mw89r\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:20.279152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.279083 5946 generic.go:356] "Generic (PLEG): container finished" podID="b5836760-c55b-46d4-9050-f33870943c0b" containerID="15705c1b2178c0d19aaf712b2fa7ca83265a74730d05c438f17af7a18a1f6503" exitCode=60 Nov 03 16:45:20.279601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.279155 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerDied","Data":"15705c1b2178c0d19aaf712b2fa7ca83265a74730d05c438f17af7a18a1f6503"} Nov 03 16:45:20.279601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.279225 5946 scope.go:122] "RemoveContainer" containerID="8deee6754777e4baa5224902fd09c1c2b969037bca45e334969127ad2ba1e658" Nov 03 16:45:20.279681 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.279633 5946 scope.go:122] "RemoveContainer" containerID="15705c1b2178c0d19aaf712b2fa7ca83265a74730d05c438f17af7a18a1f6503" Nov 03 16:45:20.279970 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:20.279951 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb_odh-ai-gateway-infra(b5836760-c55b-46d4-9050-f33870943c0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" podUID="b5836760-c55b-46d4-9050-f33870943c0b" Nov 03 16:45:20.282045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.282023 5946 generic.go:356] "Generic (PLEG): container finished" podID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerID="57cf9953d56d3eb2d2892c6fe1a2dc17bf9036cfccc1cf0df59c74f609514978" exitCode=60 Nov 03 16:45:20.282143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.282093 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerDied","Data":"57cf9953d56d3eb2d2892c6fe1a2dc17bf9036cfccc1cf0df59c74f609514978"} Nov 03 16:45:20.282484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.282466 5946 scope.go:122] "RemoveContainer" containerID="57cf9953d56d3eb2d2892c6fe1a2dc17bf9036cfccc1cf0df59c74f609514978" Nov 03 16:45:20.282816 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:20.282796 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw_odh-ai-gateway-infra(9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" Nov 03 16:45:20.284001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.283972 5946 generic.go:356] "Generic (PLEG): container finished" podID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerID="6c88dd5e4c08d92c3a28fb44f738e850e64eb351c7dbe05eb643c133f9340076" exitCode=60 Nov 03 16:45:20.284103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.284036 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerDied","Data":"6c88dd5e4c08d92c3a28fb44f738e850e64eb351c7dbe05eb643c133f9340076"} Nov 03 16:45:20.284397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.284381 5946 scope.go:122] "RemoveContainer" containerID="6c88dd5e4c08d92c3a28fb44f738e850e64eb351c7dbe05eb643c133f9340076" Nov 03 16:45:20.284717 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:20.284672 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r_odh-ai-gateway-infra(eefd3d1f-d7e6-4bef-b033-508b910c0a2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" Nov 03 16:45:20.285765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.285749 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" Nov 03 16:45:20.285854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.285766 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84" event={"ID":"ce2f4ad1-baad-41c9-a3e4-923dcae51c41","Type":"ContainerDied","Data":"81bf33e8d4be82235b91439e0b9a6d92d2aafb1f7e15957d6a032647c0135045"} Nov 03 16:45:20.287200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.287180 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" Nov 03 16:45:20.287336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.287197 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf" event={"ID":"4cd2a9eb-4be2-4031-822d-b49a4670b679","Type":"ContainerDied","Data":"03112f578453e7ecff243d7ded12c207b5b82475ec7826e8934b5b66b52d85be"} Nov 03 16:45:20.289652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.289632 5946 generic.go:356] "Generic (PLEG): container finished" podID="3de379cd-d26a-4e49-be70-f220543c177a" containerID="079547130a0e1f021c983400139640824bb0d584a9bb2fa47273ff2109947217" exitCode=60 Nov 03 16:45:20.289736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.289710 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerDied","Data":"079547130a0e1f021c983400139640824bb0d584a9bb2fa47273ff2109947217"} Nov 03 16:45:20.290057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.290041 5946 scope.go:122] "RemoveContainer" containerID="079547130a0e1f021c983400139640824bb0d584a9bb2fa47273ff2109947217" Nov 03 16:45:20.291511 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:20.290350 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz_odh-ai-gateway-infra(3de379cd-d26a-4e49-be70-f220543c177a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" podUID="3de379cd-d26a-4e49-be70-f220543c177a" Nov 03 16:45:20.292749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.292729 5946 generic.go:356] "Generic (PLEG): container finished" podID="65d096c2-13d5-417c-a2d8-498581ab250f" containerID="f5c55e03361c6c6eafa57ebf72ab3031196317a60bbbbabaf95be4f485b61ca8" exitCode=60 Nov 03 16:45:20.292850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.292807 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerDied","Data":"f5c55e03361c6c6eafa57ebf72ab3031196317a60bbbbabaf95be4f485b61ca8"} Nov 03 16:45:20.293041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.293028 5946 scope.go:122] "RemoveContainer" containerID="f5c55e03361c6c6eafa57ebf72ab3031196317a60bbbbabaf95be4f485b61ca8" Nov 03 16:45:20.293291 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:20.293270 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f_odh-ai-gateway-infra(65d096c2-13d5-417c-a2d8-498581ab250f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" Nov 03 16:45:20.294803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.294785 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" event={"ID":"ccbfe05c-7d59-4e1a-81ed-f480be5892d0","Type":"ContainerDied","Data":"942eae2fa096fb2531ba9ac247197a8b306ca6034801b6798b745191f7ec6e4e"} Nov 03 16:45:20.294875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.294860 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d" Nov 03 16:45:20.296787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.296768 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" event={"ID":"bd3a99a6-3244-414d-aaba-1ac35a4ec7a5","Type":"ContainerDied","Data":"f256651d11ca24947f5f931014de8c54861209af175b76c15f63dd38558e78f5"} Nov 03 16:45:20.296872 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.296850 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr" Nov 03 16:45:20.298873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.298856 5946 generic.go:356] "Generic (PLEG): container finished" podID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerID="852880a91f263608adcb2bc3a026cc216eb22b7f83891c94ed08d716ee2e864f" exitCode=60 Nov 03 16:45:20.298965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.298917 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" event={"ID":"70db9170-6a1c-4b21-b509-67d85e1cee91","Type":"ContainerDied","Data":"852880a91f263608adcb2bc3a026cc216eb22b7f83891c94ed08d716ee2e864f"} Nov 03 16:45:20.299274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.299258 5946 scope.go:122] "RemoveContainer" containerID="852880a91f263608adcb2bc3a026cc216eb22b7f83891c94ed08d716ee2e864f" Nov 03 16:45:20.299477 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:20.299456 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh_odh-ai-gateway-infra(70db9170-6a1c-4b21-b509-67d85e1cee91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" Nov 03 16:45:20.301360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.300947 5946 generic.go:356] "Generic (PLEG): container finished" podID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerID="230897a9b797396c55b7a8c15bf0bfdd336ac67dcc755d824353e2e6cdaf3155" exitCode=60 Nov 03 16:45:20.301360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.301014 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" event={"ID":"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7","Type":"ContainerDied","Data":"230897a9b797396c55b7a8c15bf0bfdd336ac67dcc755d824353e2e6cdaf3155"} Nov 03 16:45:20.301527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.301370 5946 scope.go:122] "RemoveContainer" containerID="230897a9b797396c55b7a8c15bf0bfdd336ac67dcc755d824353e2e6cdaf3155" Nov 03 16:45:20.301695 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:20.301631 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng_odh-ai-gateway-infra(ce39a13e-32a3-4ab3-88bc-118b76cc2ea7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" Nov 03 16:45:20.301912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.301892 5946 scope.go:122] "RemoveContainer" containerID="3662126fc8ff5253363cc14a2fce3b23e19b42e36f94c166934d1b80404d56b0" Nov 03 16:45:20.344555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.344531 5946 scope.go:122] "RemoveContainer" containerID="5fc169de0a5c7e844599b004c1c16e87edf4b592dbf1ced0b8dddcb5f13aff8f" Nov 03 16:45:20.363664 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.363639 5946 scope.go:122] "RemoveContainer" containerID="2da01c0698e7cbec36486a9dcc526a7053fb8047b702f7263125954379f8b88e" Nov 03 16:45:20.374727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.374709 5946 scope.go:122] "RemoveContainer" containerID="2bf5234a730d7c64cb841e8aebcd15ec62e64a828494d68b688dd0cc36062ad5" Nov 03 16:45:20.385033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.385014 5946 scope.go:122] "RemoveContainer" containerID="e497cb35e1f9b79a05c4d8e4d54c0175c52661fcf9387711943bb78fc68f5ba6" Nov 03 16:45:20.399914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.399888 5946 scope.go:122] "RemoveContainer" containerID="b2d7a811a62102ec57a3e23c5dd76441577e272934e1482f0e83a49af36e7686" Nov 03 16:45:20.429733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.429707 5946 scope.go:122] "RemoveContainer" containerID="58d98e23434ffd6234fe5026dc32ef8045bf47fe850c1cb22e40ab8fd646e110" Nov 03 16:45:20.455495 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.455156 5946 scope.go:122] "RemoveContainer" containerID="aba0f248bdd312f20f2e841b0f2fb32387989bb6b5491f31a5968862e3b93f9a" Nov 03 16:45:20.473402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.469970 5946 scope.go:122] "RemoveContainer" containerID="2fd02cba6a84683cebcb803ccd121029f21e88e4b05ee560a0a432afff75775a" Nov 03 16:45:20.544480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.544437 5946 scope.go:122] "RemoveContainer" containerID="74b0872b83425fabdd2759f325a1fba74d590199b4ba2bc0fde07580e1675a98" Nov 03 16:45:20.550954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.550849 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r"] Nov 03 16:45:20.747941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.747799 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f"] Nov 03 16:45:20.845102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.845013 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb"] Nov 03 16:45:20.987468 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:20.987432 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng"] Nov 03 16:45:21.244863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.244832 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw"] Nov 03 16:45:21.324897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.324868 5946 generic.go:356] "Generic (PLEG): container finished" podID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerID="695cd2befb36f792efbb79a8d7ae6d6899a6e4a6441a8fcd457af5434f088ed6" exitCode=60 Nov 03 16:45:21.325330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.324944 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" event={"ID":"7101b7e0-dfa3-4b10-b60b-e515c861f40e","Type":"ContainerDied","Data":"695cd2befb36f792efbb79a8d7ae6d6899a6e4a6441a8fcd457af5434f088ed6"} Nov 03 16:45:21.325330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.325016 5946 scope.go:122] "RemoveContainer" containerID="900bec8396b32e5c8c45f3da7f2fa9c0e408dcb664de8748cb656662fea76a3e" Nov 03 16:45:21.325573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.325553 5946 scope.go:122] "RemoveContainer" containerID="695cd2befb36f792efbb79a8d7ae6d6899a6e4a6441a8fcd457af5434f088ed6" Nov 03 16:45:21.325972 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:21.325935 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-gpl78_odh-ai-gateway-infra(7101b7e0-dfa3-4b10-b60b-e515c861f40e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" Nov 03 16:45:21.331702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.331635 5946 generic.go:356] "Generic (PLEG): container finished" podID="ba75d816-f403-4721-acc8-60bddc30be52" containerID="2c373b5e684c353be4c9a48c46a399f9c28c9db4d63546360d252ccea9fed067" exitCode=60 Nov 03 16:45:21.331702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.331687 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" event={"ID":"ba75d816-f403-4721-acc8-60bddc30be52","Type":"ContainerDied","Data":"2c373b5e684c353be4c9a48c46a399f9c28c9db4d63546360d252ccea9fed067"} Nov 03 16:45:21.332569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.332280 5946 scope.go:122] "RemoveContainer" containerID="2c373b5e684c353be4c9a48c46a399f9c28c9db4d63546360d252ccea9fed067" Nov 03 16:45:21.332724 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:21.332694 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7_odh-ai-gateway-infra(ba75d816-f403-4721-acc8-60bddc30be52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" podUID="ba75d816-f403-4721-acc8-60bddc30be52" Nov 03 16:45:21.377757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.377726 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz"] Nov 03 16:45:21.378055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.377809 5946 scope.go:122] "RemoveContainer" containerID="76895f50bb507da3d4f6410c13ff10fbbd43a53fe5e8d001685e673b2689340f" Nov 03 16:45:21.587854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.587505 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:21.660581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.660182 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5836760-c55b-46d4-9050-f33870943c0b-kube-api-access-wwpnb\" (UniqueName: \"kubernetes.io/projected/b5836760-c55b-46d4-9050-f33870943c0b-kube-api-access-wwpnb\") pod \"b5836760-c55b-46d4-9050-f33870943c0b\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " Nov 03 16:45:21.661452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.661411 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5836760-c55b-46d4-9050-f33870943c0b-maas-api-service-ca" pod "b5836760-c55b-46d4-9050-f33870943c0b" (UID: "b5836760-c55b-46d4-9050-f33870943c0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:21.661609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.661591 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5836760-c55b-46d4-9050-f33870943c0b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5836760-c55b-46d4-9050-f33870943c0b-maas-api-service-ca\") pod \"b5836760-c55b-46d4-9050-f33870943c0b\" (UID: \"b5836760-c55b-46d4-9050-f33870943c0b\") " Nov 03 16:45:21.662413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.662393 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5836760-c55b-46d4-9050-f33870943c0b-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.672705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.672647 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf"] Nov 03 16:45:21.692997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.692962 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5937b-29895405-xt4mf"] Nov 03 16:45:21.697419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.697321 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5836760-c55b-46d4-9050-f33870943c0b-kube-api-access-wwpnb" pod "b5836760-c55b-46d4-9050-f33870943c0b" (UID: "b5836760-c55b-46d4-9050-f33870943c0b"). InnerVolumeSpecName "kube-api-access-wwpnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:21.701514 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.701449 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:21.739922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.739891 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:21.742153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.742114 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:21.745444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.745413 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.758887 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.764323 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de379cd-d26a-4e49-be70-f220543c177a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de379cd-d26a-4e49-be70-f220543c177a-maas-api-service-ca\") pod \"3de379cd-d26a-4e49-be70-f220543c177a\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.764595 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-maas-api-service-ca\") pod \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.764649 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-kube-api-access-w2sth\" (UniqueName: \"kubernetes.io/projected/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-kube-api-access-w2sth\") pod \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.764679 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d096c2-13d5-417c-a2d8-498581ab250f-kube-api-access-bv2qm\" (UniqueName: \"kubernetes.io/projected/65d096c2-13d5-417c-a2d8-498581ab250f-kube-api-access-bv2qm\") pod \"65d096c2-13d5-417c-a2d8-498581ab250f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.764760 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-kube-api-access-s8gpf\" (UniqueName: \"kubernetes.io/projected/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-kube-api-access-s8gpf\") pod \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\" (UID: \"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.764842 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d096c2-13d5-417c-a2d8-498581ab250f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d096c2-13d5-417c-a2d8-498581ab250f-maas-api-service-ca\") pod \"65d096c2-13d5-417c-a2d8-498581ab250f\" (UID: \"65d096c2-13d5-417c-a2d8-498581ab250f\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.764876 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-maas-api-service-ca\") pod \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\" (UID: \"eefd3d1f-d7e6-4bef-b033-508b910c0a2d\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.779419 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.780679 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-maas-api-service-ca" pod "9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" (UID: "9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.792957 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de379cd-d26a-4e49-be70-f220543c177a-kube-api-access-f4smb\" (UniqueName: \"kubernetes.io/projected/3de379cd-d26a-4e49-be70-f220543c177a-kube-api-access-f4smb\") pod \"3de379cd-d26a-4e49-be70-f220543c177a\" (UID: \"3de379cd-d26a-4e49-be70-f220543c177a\") " Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.793870 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d096c2-13d5-417c-a2d8-498581ab250f-maas-api-service-ca" pod "65d096c2-13d5-417c-a2d8-498581ab250f" (UID: "65d096c2-13d5-417c-a2d8-498581ab250f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.794395 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-maas-api-service-ca" pod "eefd3d1f-d7e6-4bef-b033-508b910c0a2d" (UID: "eefd3d1f-d7e6-4bef-b033-508b910c0a2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.794598 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de379cd-d26a-4e49-be70-f220543c177a-maas-api-service-ca" pod "3de379cd-d26a-4e49-be70-f220543c177a" (UID: "3de379cd-d26a-4e49-be70-f220543c177a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.795071 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d096c2-13d5-417c-a2d8-498581ab250f-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.795099 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.795114 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de379cd-d26a-4e49-be70-f220543c177a-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.795129 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwpnb\" (UniqueName: \"kubernetes.io/projected/b5836760-c55b-46d4-9050-f33870943c0b-kube-api-access-wwpnb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.795145 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.801486 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-kube-api-access-s8gpf" pod "9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" (UID: "9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44"). InnerVolumeSpecName "kube-api-access-s8gpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.801954 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-kube-api-access-w2sth" pod "eefd3d1f-d7e6-4bef-b033-508b910c0a2d" (UID: "eefd3d1f-d7e6-4bef-b033-508b910c0a2d"). InnerVolumeSpecName "kube-api-access-w2sth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.803189 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d096c2-13d5-417c-a2d8-498581ab250f-kube-api-access-bv2qm" pod "65d096c2-13d5-417c-a2d8-498581ab250f" (UID: "65d096c2-13d5-417c-a2d8-498581ab250f"). InnerVolumeSpecName "kube-api-access-bv2qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:21.806292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.804712 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de379cd-d26a-4e49-be70-f220543c177a-kube-api-access-f4smb" pod "3de379cd-d26a-4e49-be70-f220543c177a" (UID: "3de379cd-d26a-4e49-be70-f220543c177a"). InnerVolumeSpecName "kube-api-access-f4smb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:21.897537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.896413 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-maas-api-service-ca\") pod \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " Nov 03 16:45:21.897537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.896540 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-kube-api-access-fkqbs\" (UniqueName: \"kubernetes.io/projected/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-kube-api-access-fkqbs\") pod \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\" (UID: \"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7\") " Nov 03 16:45:21.897537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.897413 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2sth\" (UniqueName: \"kubernetes.io/projected/eefd3d1f-d7e6-4bef-b033-508b910c0a2d-kube-api-access-w2sth\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.897537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.897438 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2qm\" (UniqueName: \"kubernetes.io/projected/65d096c2-13d5-417c-a2d8-498581ab250f-kube-api-access-bv2qm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.897537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.897453 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gpf\" (UniqueName: \"kubernetes.io/projected/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44-kube-api-access-s8gpf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.897537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.897476 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4smb\" (UniqueName: \"kubernetes.io/projected/3de379cd-d26a-4e49-be70-f220543c177a-kube-api-access-f4smb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.898519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.898457 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-maas-api-service-ca" pod "ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" (UID: "ce39a13e-32a3-4ab3-88bc-118b76cc2ea7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:21.900543 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.900515 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-kube-api-access-fkqbs" pod "ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" (UID: "ce39a13e-32a3-4ab3-88bc-118b76cc2ea7"). InnerVolumeSpecName "kube-api-access-fkqbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:21.909194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.909135 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr"] Nov 03 16:45:21.919493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.919472 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cc4dcd-29895405-59ljr"] Nov 03 16:45:21.998812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.998778 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:21.998812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:21.998808 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqbs\" (UniqueName: \"kubernetes.io/projected/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7-kube-api-access-fkqbs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:22.006803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.006774 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" path="/var/lib/kubelet/pods/4cd2a9eb-4be2-4031-822d-b49a4670b679/volumes" Nov 03 16:45:22.007315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.007277 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" path="/var/lib/kubelet/pods/bd3a99a6-3244-414d-aaba-1ac35a4ec7a5/volumes" Nov 03 16:45:22.035518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.035478 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d"] Nov 03 16:45:22.054491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.040711 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-509f14-29895405-h4b5d"] Nov 03 16:45:22.244761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.244730 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh"] Nov 03 16:45:22.307873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.307841 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84"] Nov 03 16:45:22.322179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.322134 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-547a38-29895405-wqn84"] Nov 03 16:45:22.350718 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.350678 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" event={"ID":"ce39a13e-32a3-4ab3-88bc-118b76cc2ea7","Type":"ContainerDied","Data":"4bdf6fe4bc671ce569b0277a8dd6c7bd0dc2a1382e52e8803731f3bd6943e893"} Nov 03 16:45:22.351143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.350754 5946 scope.go:122] "RemoveContainer" containerID="230897a9b797396c55b7a8c15bf0bfdd336ac67dcc755d824353e2e6cdaf3155" Nov 03 16:45:22.353680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.351799 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng" Nov 03 16:45:22.353680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.352875 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" event={"ID":"b5836760-c55b-46d4-9050-f33870943c0b","Type":"ContainerDied","Data":"4b17f043a63ae58e2804cb42e59e9830023f09e31badf3ed205d7fa552e8ea94"} Nov 03 16:45:22.353680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.352970 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb" Nov 03 16:45:22.357379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.357355 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" event={"ID":"9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44","Type":"ContainerDied","Data":"2af1048226c704c6bd06c4226bf45231f8dc2498f814a98c68075409bf683fed"} Nov 03 16:45:22.357547 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.357441 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw" Nov 03 16:45:22.362951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.362933 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" Nov 03 16:45:22.363120 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.363093 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r" event={"ID":"eefd3d1f-d7e6-4bef-b033-508b910c0a2d","Type":"ContainerDied","Data":"a43f960331f477a4c6811c9c0b1e2e8c34fe66d84a3d6d1339e6f69869dbf447"} Nov 03 16:45:22.366881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.365127 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" event={"ID":"3de379cd-d26a-4e49-be70-f220543c177a","Type":"ContainerDied","Data":"ecb4d00fa2b50c30e0ff31178ef145e593c31811c14306090cccd1e138461cb7"} Nov 03 16:45:22.366881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.365287 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz" Nov 03 16:45:22.372899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.372882 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" event={"ID":"65d096c2-13d5-417c-a2d8-498581ab250f","Type":"ContainerDied","Data":"b2a6cb0ab6b8efb27ed462e1a9d973e43270171ed2c2e2d407d06d2b19f45d50"} Nov 03 16:45:22.373126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.372893 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f" Nov 03 16:45:22.375892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.375817 5946 scope.go:122] "RemoveContainer" containerID="15705c1b2178c0d19aaf712b2fa7ca83265a74730d05c438f17af7a18a1f6503" Nov 03 16:45:22.387291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.387263 5946 scope.go:122] "RemoveContainer" containerID="57cf9953d56d3eb2d2892c6fe1a2dc17bf9036cfccc1cf0df59c74f609514978" Nov 03 16:45:22.398054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.398035 5946 scope.go:122] "RemoveContainer" containerID="6c88dd5e4c08d92c3a28fb44f738e850e64eb351c7dbe05eb643c133f9340076" Nov 03 16:45:22.399844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.399825 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:22.409845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.409816 5946 scope.go:122] "RemoveContainer" containerID="079547130a0e1f021c983400139640824bb0d584a9bb2fa47273ff2109947217" Nov 03 16:45:22.419258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.419228 5946 scope.go:122] "RemoveContainer" containerID="f5c55e03361c6c6eafa57ebf72ab3031196317a60bbbbabaf95be4f485b61ca8" Nov 03 16:45:22.578315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.578283 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70db9170-6a1c-4b21-b509-67d85e1cee91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70db9170-6a1c-4b21-b509-67d85e1cee91-maas-api-service-ca\") pod \"70db9170-6a1c-4b21-b509-67d85e1cee91\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " Nov 03 16:45:22.578587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.578565 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70db9170-6a1c-4b21-b509-67d85e1cee91-kube-api-access-8lvnz\" (UniqueName: \"kubernetes.io/projected/70db9170-6a1c-4b21-b509-67d85e1cee91-kube-api-access-8lvnz\") pod \"70db9170-6a1c-4b21-b509-67d85e1cee91\" (UID: \"70db9170-6a1c-4b21-b509-67d85e1cee91\") " Nov 03 16:45:22.629308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.619408 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70db9170-6a1c-4b21-b509-67d85e1cee91-maas-api-service-ca" pod "70db9170-6a1c-4b21-b509-67d85e1cee91" (UID: "70db9170-6a1c-4b21-b509-67d85e1cee91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:22.641152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.641084 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70db9170-6a1c-4b21-b509-67d85e1cee91-kube-api-access-8lvnz" pod "70db9170-6a1c-4b21-b509-67d85e1cee91" (UID: "70db9170-6a1c-4b21-b509-67d85e1cee91"). InnerVolumeSpecName "kube-api-access-8lvnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:22.680256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.680223 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70db9170-6a1c-4b21-b509-67d85e1cee91-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:22.680256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:22.680257 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvnz\" (UniqueName: \"kubernetes.io/projected/70db9170-6a1c-4b21-b509-67d85e1cee91-kube-api-access-8lvnz\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:23.280858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.280807 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:23.386078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.386046 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" Nov 03 16:45:23.386595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.386041 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh" event={"ID":"70db9170-6a1c-4b21-b509-67d85e1cee91","Type":"ContainerDied","Data":"aaf6d4a0228fdcfbccd7d33c096d250a67453b3ea5f24dece8876cea63b070ee"} Nov 03 16:45:23.386595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.386201 5946 scope.go:122] "RemoveContainer" containerID="852880a91f263608adcb2bc3a026cc216eb22b7f83891c94ed08d716ee2e864f" Nov 03 16:45:23.496359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.496247 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:23.543769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.543658 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r"] Nov 03 16:45:23.548254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.548183 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895405-njb8r"] Nov 03 16:45:23.666753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.666712 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz"] Nov 03 16:45:23.672838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.672807 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895405-vmmwz"] Nov 03 16:45:23.683244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.683175 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:23.755868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.755833 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl"] Nov 03 16:45:23.761041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.761010 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-438248-j8vfl"] Nov 03 16:45:23.825778 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.825665 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7"] Nov 03 16:45:23.854452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.854416 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh"] Nov 03 16:45:23.864872 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.864839 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2537c1-29895405-zs9dh"] Nov 03 16:45:23.878102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.878060 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:23.915010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.914966 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:45:23.915184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.915037 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:45:23.956577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.956541 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78"] Nov 03 16:45:23.981020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.980988 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f"] Nov 03 16:45:23.987359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:23.987331 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-192564-29895405-75w8f"] Nov 03 16:45:24.012258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.012150 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:24.016255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.016225 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de379cd-d26a-4e49-be70-f220543c177a" path="/var/lib/kubelet/pods/3de379cd-d26a-4e49-be70-f220543c177a/volumes" Nov 03 16:45:24.016790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.016769 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" path="/var/lib/kubelet/pods/65d096c2-13d5-417c-a2d8-498581ab250f/volumes" Nov 03 16:45:24.017269 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.017250 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" path="/var/lib/kubelet/pods/70db9170-6a1c-4b21-b509-67d85e1cee91/volumes" Nov 03 16:45:24.018049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.018030 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4" path="/var/lib/kubelet/pods/8d11b7fc-8ad4-403e-90c9-2c1087b4a9f4/volumes" Nov 03 16:45:24.018501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.018484 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" path="/var/lib/kubelet/pods/ccbfe05c-7d59-4e1a-81ed-f480be5892d0/volumes" Nov 03 16:45:24.018912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.018894 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" path="/var/lib/kubelet/pods/ce2f4ad1-baad-41c9-a3e4-923dcae51c41/volumes" Nov 03 16:45:24.019633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.019613 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" path="/var/lib/kubelet/pods/eefd3d1f-d7e6-4bef-b033-508b910c0a2d/volumes" Nov 03 16:45:24.108162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.106564 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb"] Nov 03 16:45:24.109186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.108348 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba75d816-f403-4721-acc8-60bddc30be52-kube-api-access-jhtf8\" (UniqueName: \"kubernetes.io/projected/ba75d816-f403-4721-acc8-60bddc30be52-kube-api-access-jhtf8\") pod \"ba75d816-f403-4721-acc8-60bddc30be52\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " Nov 03 16:45:24.109186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.108432 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba75d816-f403-4721-acc8-60bddc30be52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba75d816-f403-4721-acc8-60bddc30be52-maas-api-service-ca\") pod \"ba75d816-f403-4721-acc8-60bddc30be52\" (UID: \"ba75d816-f403-4721-acc8-60bddc30be52\") " Nov 03 16:45:24.113048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.110449 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba75d816-f403-4721-acc8-60bddc30be52-maas-api-service-ca" pod "ba75d816-f403-4721-acc8-60bddc30be52" (UID: "ba75d816-f403-4721-acc8-60bddc30be52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:24.113048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.111036 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba75d816-f403-4721-acc8-60bddc30be52-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:24.113048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.111699 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba75d816-f403-4721-acc8-60bddc30be52-kube-api-access-jhtf8" pod "ba75d816-f403-4721-acc8-60bddc30be52" (UID: "ba75d816-f403-4721-acc8-60bddc30be52"). InnerVolumeSpecName "kube-api-access-jhtf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:24.113336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.113307 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895405-nbjlb"] Nov 03 16:45:24.147112 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.147086 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:24.203906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.203873 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:24.212238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.212195 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7101b7e0-dfa3-4b10-b60b-e515c861f40e-kube-api-access-qw5v5\" (UniqueName: \"kubernetes.io/projected/7101b7e0-dfa3-4b10-b60b-e515c861f40e-kube-api-access-qw5v5\") pod \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " Nov 03 16:45:24.212374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.212359 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7101b7e0-dfa3-4b10-b60b-e515c861f40e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7101b7e0-dfa3-4b10-b60b-e515c861f40e-maas-api-service-ca\") pod \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\" (UID: \"7101b7e0-dfa3-4b10-b60b-e515c861f40e\") " Nov 03 16:45:24.212695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.212668 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7101b7e0-dfa3-4b10-b60b-e515c861f40e-maas-api-service-ca" pod "7101b7e0-dfa3-4b10-b60b-e515c861f40e" (UID: "7101b7e0-dfa3-4b10-b60b-e515c861f40e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:24.212916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.212900 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtf8\" (UniqueName: \"kubernetes.io/projected/ba75d816-f403-4721-acc8-60bddc30be52-kube-api-access-jhtf8\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:24.212998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.212917 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7101b7e0-dfa3-4b10-b60b-e515c861f40e-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:24.214636 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.214611 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7101b7e0-dfa3-4b10-b60b-e515c861f40e-kube-api-access-qw5v5" pod "7101b7e0-dfa3-4b10-b60b-e515c861f40e" (UID: "7101b7e0-dfa3-4b10-b60b-e515c861f40e"). InnerVolumeSpecName "kube-api-access-qw5v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:24.225782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.225753 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng"] Nov 03 16:45:24.233879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.233849 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9635db0d-29895405-t74ng"] Nov 03 16:45:24.314802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.314757 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw5v5\" (UniqueName: \"kubernetes.io/projected/7101b7e0-dfa3-4b10-b60b-e515c861f40e-kube-api-access-qw5v5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:24.348138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.348095 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw"] Nov 03 16:45:24.355757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.355729 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cb494a-29895405-jlmzw"] Nov 03 16:45:24.401956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.401882 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" Nov 03 16:45:24.401956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.401919 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78" event={"ID":"7101b7e0-dfa3-4b10-b60b-e515c861f40e","Type":"ContainerDied","Data":"ae0158fc86c6b4df5626f40f9b74c942a8b27cc410a4959c3fe6efbb8ae11e5b"} Nov 03 16:45:24.402518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.402003 5946 scope.go:122] "RemoveContainer" containerID="695cd2befb36f792efbb79a8d7ae6d6899a6e4a6441a8fcd457af5434f088ed6" Nov 03 16:45:24.403744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.403708 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" Nov 03 16:45:24.403970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.403713 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7" event={"ID":"ba75d816-f403-4721-acc8-60bddc30be52","Type":"ContainerDied","Data":"e36014e90abdfc484f1f6ddf7139ba5a514b56a1070f86a97553e72ed7cb6e3f"} Nov 03 16:45:24.414236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.414165 5946 scope.go:122] "RemoveContainer" containerID="2c373b5e684c353be4c9a48c46a399f9c28c9db4d63546360d252ccea9fed067" Nov 03 16:45:24.707611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.707515 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7"] Nov 03 16:45:24.710650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.710621 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-66f145-29895405-7lvk7"] Nov 03 16:45:24.823058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.823028 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78"] Nov 03 16:45:24.830461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:24.830428 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-gpl78"] Nov 03 16:45:25.419479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:25.419441 5946 generic.go:356] "Generic (PLEG): container finished" podID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerID="395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299" exitCode=0 Nov 03 16:45:25.420036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:25.419518 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" event={"ID":"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e","Type":"ContainerDied","Data":"395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299"} Nov 03 16:45:25.420114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:25.420058 5946 scope.go:122] "RemoveContainer" containerID="395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299" Nov 03 16:45:25.422103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:25.422067 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:45:25.422246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:25.422139 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:45:25.422246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:25.422172 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:45:26.008906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.008873 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" path="/var/lib/kubelet/pods/7101b7e0-dfa3-4b10-b60b-e515c861f40e/volumes" Nov 03 16:45:26.009358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.009344 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" path="/var/lib/kubelet/pods/9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44/volumes" Nov 03 16:45:26.009753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.009739 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5836760-c55b-46d4-9050-f33870943c0b" path="/var/lib/kubelet/pods/b5836760-c55b-46d4-9050-f33870943c0b/volumes" Nov 03 16:45:26.010432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.010418 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba75d816-f403-4721-acc8-60bddc30be52" path="/var/lib/kubelet/pods/ba75d816-f403-4721-acc8-60bddc30be52/volumes" Nov 03 16:45:26.010811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.010799 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" path="/var/lib/kubelet/pods/ce39a13e-32a3-4ab3-88bc-118b76cc2ea7/volumes" Nov 03 16:45:26.341107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341014 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f"] Nov 03 16:45:26.341742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341720 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:26.341794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341744 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:26.341794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341753 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:26.341794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341758 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:26.341794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341770 5946 cpu_manager.go:414] "containerMap: removing container" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:26.341794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341775 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:26.341794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341786 5946 cpu_manager.go:414] "containerMap: removing container" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:26.341794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341791 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341801 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341809 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341817 5946 cpu_manager.go:414] "containerMap: removing container" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341822 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341833 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341837 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341844 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341849 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341856 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341860 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341873 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341880 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341890 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341897 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341905 5946 cpu_manager.go:414] "containerMap: removing container" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341910 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341917 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341921 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341929 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341934 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341942 5946 cpu_manager.go:414] "containerMap: removing container" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341946 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341958 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341966 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341974 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341979 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341989 5946 cpu_manager.go:414] "containerMap: removing container" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.341992 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342000 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342004 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342009 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342013 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342020 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342024 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:26.342018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342032 5946 cpu_manager.go:414] "containerMap: removing container" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342039 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342047 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342051 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342058 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342062 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342072 5946 cpu_manager.go:414] "containerMap: removing container" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342077 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342084 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342087 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342095 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342099 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342107 5946 cpu_manager.go:414] "containerMap: removing container" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342113 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342121 5946 cpu_manager.go:414] "containerMap: removing container" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342125 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342132 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342136 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342144 5946 cpu_manager.go:414] "containerMap: removing container" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342149 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342264 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342276 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342282 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342289 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342295 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342301 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342306 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342313 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342319 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342323 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342329 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342334 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342340 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342348 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342356 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342360 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342367 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342372 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342377 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342384 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342389 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342394 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342400 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342405 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342411 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342418 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d27bf2c-ea47-4a96-931c-bbf2c2ba0d44" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342429 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342434 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342440 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342446 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342451 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342456 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:26.343113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.342460 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:26.348399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.348376 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.409121 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.409035 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f"] Nov 03 16:45:26.459239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.459104 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:45:26.460256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.460014 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:45:26.460256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.460093 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:45:26.460256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.460174 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:45:26.551130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.550324 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" event={"ID":"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e","Type":"ContainerStarted","Data":"b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713"} Nov 03 16:45:26.568517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.568474 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdfpn\" (UniqueName: \"kubernetes.io/projected/b6f4e9f8-6817-4913-b142-4b68439cd2d5-kube-api-access-fdfpn\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.568734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.568643 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.568797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.568747 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.669704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.669606 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfpn\" (UniqueName: \"kubernetes.io/projected/b6f4e9f8-6817-4913-b142-4b68439cd2d5-kube-api-access-fdfpn\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.669886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.669730 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.669886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.669824 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.670628 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.670597 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.672751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.672727 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.743042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.743002 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfpn\" (UniqueName: \"kubernetes.io/projected/b6f4e9f8-6817-4913-b142-4b68439cd2d5-kube-api-access-fdfpn\") pod \"maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:26.961929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:26.961852 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:27.264989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:27.264959 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:45:27.265160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:27.265139 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 16:45:27.265235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:27.265185 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 16:45:27.371180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:27.371143 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f"] Nov 03 16:45:27.372480 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:27.372452 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f4e9f8_6817_4913_b142_4b68439cd2d5.slice/crio-45a0d09a8eed3d65a2632ecfa71d54677b8e2e987e55c028f7838e70d18bef09 WatchSource:0}: Error finding container 45a0d09a8eed3d65a2632ecfa71d54677b8e2e987e55c028f7838e70d18bef09: Status 404 returned error can't find the container with id 45a0d09a8eed3d65a2632ecfa71d54677b8e2e987e55c028f7838e70d18bef09 Nov 03 16:45:27.560022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:27.559984 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" event={"ID":"b6f4e9f8-6817-4913-b142-4b68439cd2d5","Type":"ContainerStarted","Data":"45a0d09a8eed3d65a2632ecfa71d54677b8e2e987e55c028f7838e70d18bef09"} Nov 03 16:45:28.265327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:28.265289 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 16:45:28.265528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:28.265344 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 16:45:28.569408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:28.569311 5946 generic.go:356] "Generic (PLEG): container finished" podID="b6f4e9f8-6817-4913-b142-4b68439cd2d5" containerID="5a14862a6d54b17bd0ac1e13e2a4057845c09642dc68f9cbe1238d9b0c73b586" exitCode=0 Nov 03 16:45:28.569408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:28.569391 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" event={"ID":"b6f4e9f8-6817-4913-b142-4b68439cd2d5","Type":"ContainerDied","Data":"5a14862a6d54b17bd0ac1e13e2a4057845c09642dc68f9cbe1238d9b0c73b586"} Nov 03 16:45:28.970457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:28.968340 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:29.265992 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.265949 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 16:45:29.266257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.266011 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 16:45:29.781080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.781051 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:29.812922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.812893 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f4e9f8-6817-4913-b142-4b68439cd2d5-kube-api-access-fdfpn\" (UniqueName: \"kubernetes.io/projected/b6f4e9f8-6817-4913-b142-4b68439cd2d5-kube-api-access-fdfpn\") pod \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " Nov 03 16:45:29.813123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.813109 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-xks-ca\") pod \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " Nov 03 16:45:29.813177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.813140 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-openshift-ca\") pod \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\" (UID: \"b6f4e9f8-6817-4913-b142-4b68439cd2d5\") " Nov 03 16:45:29.838781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.838590 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-openshift-ca" pod "b6f4e9f8-6817-4913-b142-4b68439cd2d5" (UID: "b6f4e9f8-6817-4913-b142-4b68439cd2d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:29.840660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.840592 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f4e9f8-6817-4913-b142-4b68439cd2d5-kube-api-access-fdfpn" pod "b6f4e9f8-6817-4913-b142-4b68439cd2d5" (UID: "b6f4e9f8-6817-4913-b142-4b68439cd2d5"). InnerVolumeSpecName "kube-api-access-fdfpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:29.840825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.840590 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-xks-ca" pod "b6f4e9f8-6817-4913-b142-4b68439cd2d5" (UID: "b6f4e9f8-6817-4913-b142-4b68439cd2d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:29.944282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.943350 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:29.944282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.943392 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f4e9f8-6817-4913-b142-4b68439cd2d5-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:29.944282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:29.943411 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdfpn\" (UniqueName: \"kubernetes.io/projected/b6f4e9f8-6817-4913-b142-4b68439cd2d5-kube-api-access-fdfpn\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:30.265339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:30.265296 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 16:45:30.265536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:30.265349 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 16:45:30.585598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:30.585518 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" Nov 03 16:45:30.585598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:30.585560 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f" event={"ID":"b6f4e9f8-6817-4913-b142-4b68439cd2d5","Type":"ContainerDied","Data":"45a0d09a8eed3d65a2632ecfa71d54677b8e2e987e55c028f7838e70d18bef09"} Nov 03 16:45:30.585813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:30.585609 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a0d09a8eed3d65a2632ecfa71d54677b8e2e987e55c028f7838e70d18bef09" Nov 03 16:45:31.264985 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:31.264952 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 16:45:31.265450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:31.265025 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 16:45:32.270139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:32.270108 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:45:32.270559 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:32.270391 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:45:32.271134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:32.271113 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:45:32.722606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:32.722520 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:32.906726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:32.906695 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:33.108260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:33.107804 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:34.004069 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.003993 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:34.220011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.219964 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:34.426569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.426467 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:34.428541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.428518 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4"] Nov 03 16:45:34.429134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429120 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:34.429183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429137 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba75d816-f403-4721-acc8-60bddc30be52" containerName="cleanup" Nov 03 16:45:34.429183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429148 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:34.429183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429153 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:34.429183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429169 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:34.429183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429174 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:34.429183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429184 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429189 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5836760-c55b-46d4-9050-f33870943c0b" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429221 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b6f4e9f8-6817-4913-b142-4b68439cd2d5" containerName="revoke-keys" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429227 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f4e9f8-6817-4913-b142-4b68439cd2d5" containerName="revoke-keys" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429244 5946 cpu_manager.go:414] "containerMap: removing container" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429249 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429263 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429268 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429274 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429278 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429287 5946 cpu_manager.go:414] "containerMap: removing container" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429292 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429300 5946 cpu_manager.go:414] "containerMap: removing container" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429303 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70db9170-6a1c-4b21-b509-67d85e1cee91" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429312 5946 cpu_manager.go:414] "containerMap: removing container" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429316 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3a99a6-3244-414d-aaba-1ac35a4ec7a5" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429324 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:34.429383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429328 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429425 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce39a13e-32a3-4ab3-88bc-118b76cc2ea7" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429434 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7101b7e0-dfa3-4b10-b60b-e515c861f40e" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429441 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d096c2-13d5-417c-a2d8-498581ab250f" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429447 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefd3d1f-d7e6-4bef-b033-508b910c0a2d" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429454 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de379cd-d26a-4e49-be70-f220543c177a" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429460 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd2a9eb-4be2-4031-822d-b49a4670b679" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429475 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82edb20-138b-4058-8303-e6b1655bc37e" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429484 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbfe05c-7d59-4e1a-81ed-f480be5892d0" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429491 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2f4ad1-baad-41c9-a3e4-923dcae51c41" containerName="cleanup" Nov 03 16:45:34.429873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.429499 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f4e9f8-6817-4913-b142-4b68439cd2d5" containerName="revoke-keys" Nov 03 16:45:34.434345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.434326 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.493145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.493116 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4"] Nov 03 16:45:34.515031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.514982 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.515253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.515193 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2v6\" (UniqueName: \"kubernetes.io/projected/85d391ad-12ba-4e68-baec-b5b212f9723d-kube-api-access-rd2v6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.515348 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.515327 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.540129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.540098 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:45:34.540306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.540129 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:45:34.540306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.540146 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:45:34.540306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.540129 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:45:34.616782 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.616752 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.616975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.616834 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2v6\" (UniqueName: \"kubernetes.io/projected/85d391ad-12ba-4e68-baec-b5b212f9723d-kube-api-access-rd2v6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.616975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.616924 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.617595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.617577 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.619308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.619289 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.677270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.677184 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2v6\" (UniqueName: \"kubernetes.io/projected/85d391ad-12ba-4e68-baec-b5b212f9723d-kube-api-access-rd2v6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:34.748430 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:34.748394 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:35.138031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:35.138006 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4"] Nov 03 16:45:35.139105 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:45:35.139078 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d391ad_12ba_4e68_baec_b5b212f9723d.slice/crio-a96e5435bf59c1b15b0407fa4279dc6f60eb9270a7330e30d9060a8eaf3e3120 WatchSource:0}: Error finding container a96e5435bf59c1b15b0407fa4279dc6f60eb9270a7330e30d9060a8eaf3e3120: Status 404 returned error can't find the container with id a96e5435bf59c1b15b0407fa4279dc6f60eb9270a7330e30d9060a8eaf3e3120 Nov 03 16:45:35.627821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:35.627787 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" event={"ID":"85d391ad-12ba-4e68-baec-b5b212f9723d","Type":"ContainerStarted","Data":"ff1c7fd5f790fa53362d8f2a0cd0c03b21f964e47ea5947442b297cb9d2786c9"} Nov 03 16:45:35.627973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:35.627842 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" event={"ID":"85d391ad-12ba-4e68-baec-b5b212f9723d","Type":"ContainerStarted","Data":"a96e5435bf59c1b15b0407fa4279dc6f60eb9270a7330e30d9060a8eaf3e3120"} Nov 03 16:45:35.749896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:35.749839 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" podStartSLOduration=1.74982507 podStartE2EDuration="1.74982507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:35.749039084 +0000 UTC m=+2005.870091910" watchObservedRunningTime="2026-11-03 16:45:35.74982507 +0000 UTC m=+2005.870877936" Nov 03 16:45:35.820188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:35.820153 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:36.636447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:36.636413 5946 generic.go:356] "Generic (PLEG): container finished" podID="85d391ad-12ba-4e68-baec-b5b212f9723d" containerID="ff1c7fd5f790fa53362d8f2a0cd0c03b21f964e47ea5947442b297cb9d2786c9" exitCode=0 Nov 03 16:45:36.636855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:36.636471 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" event={"ID":"85d391ad-12ba-4e68-baec-b5b212f9723d","Type":"ContainerDied","Data":"ff1c7fd5f790fa53362d8f2a0cd0c03b21f964e47ea5947442b297cb9d2786c9"} Nov 03 16:45:37.604144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.604109 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:37.796041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.796004 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:37.848957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.848930 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:37.973642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.973553 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-xks-ca\") pod \"85d391ad-12ba-4e68-baec-b5b212f9723d\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " Nov 03 16:45:37.973642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.973625 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-openshift-ca\") pod \"85d391ad-12ba-4e68-baec-b5b212f9723d\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " Nov 03 16:45:37.973874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.973842 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d391ad-12ba-4e68-baec-b5b212f9723d-kube-api-access-rd2v6\" (UniqueName: \"kubernetes.io/projected/85d391ad-12ba-4e68-baec-b5b212f9723d-kube-api-access-rd2v6\") pod \"85d391ad-12ba-4e68-baec-b5b212f9723d\" (UID: \"85d391ad-12ba-4e68-baec-b5b212f9723d\") " Nov 03 16:45:37.974036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.973985 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-openshift-ca" pod "85d391ad-12ba-4e68-baec-b5b212f9723d" (UID: "85d391ad-12ba-4e68-baec-b5b212f9723d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:37.974475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.974449 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:37.976920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.976885 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-xks-ca" pod "85d391ad-12ba-4e68-baec-b5b212f9723d" (UID: "85d391ad-12ba-4e68-baec-b5b212f9723d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:37.977195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:37.977162 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d391ad-12ba-4e68-baec-b5b212f9723d-kube-api-access-rd2v6" pod "85d391ad-12ba-4e68-baec-b5b212f9723d" (UID: "85d391ad-12ba-4e68-baec-b5b212f9723d"). InnerVolumeSpecName "kube-api-access-rd2v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:38.076023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:38.075992 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2v6\" (UniqueName: \"kubernetes.io/projected/85d391ad-12ba-4e68-baec-b5b212f9723d-kube-api-access-rd2v6\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:38.076023 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:38.076022 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d391ad-12ba-4e68-baec-b5b212f9723d-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:45:38.670324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:38.670293 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" event={"ID":"85d391ad-12ba-4e68-baec-b5b212f9723d","Type":"ContainerDied","Data":"a96e5435bf59c1b15b0407fa4279dc6f60eb9270a7330e30d9060a8eaf3e3120"} Nov 03 16:45:38.670500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:38.670330 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4" Nov 03 16:45:38.670500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:38.670343 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a96e5435bf59c1b15b0407fa4279dc6f60eb9270a7330e30d9060a8eaf3e3120" Nov 03 16:45:39.497291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:39.486754 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:42.587297 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:45:42.587256 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:45:44.607753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:44.607674 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:44.815535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:44.815504 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:45.008335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:45.008296 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:46.077828 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:46.077778 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:49.312493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:49.312455 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:49.510000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:49.507994 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:49.918834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:49.918794 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:54.205735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:54.205695 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:54.829850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:54.829817 5946 generic.go:356] "Generic (PLEG): container finished" podID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerID="df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70" exitCode=0 Nov 03 16:45:54.830068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:54.829881 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" event={"ID":"10e436b7-0ecc-45f2-9418-d110fadc4e67","Type":"ContainerDied","Data":"df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70"} Nov 03 16:45:54.830341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:54.830327 5946 scope.go:122] "RemoveContainer" containerID="df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70" Nov 03 16:45:54.832072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:54.832048 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:45:54.832135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:54.832118 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:45:54.832176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:54.832152 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:45:55.763826 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:55.763791 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:55.839788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:55.839751 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" event={"ID":"10e436b7-0ecc-45f2-9418-d110fadc4e67","Type":"ContainerStarted","Data":"61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3"} Nov 03 16:45:55.952233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:55.952173 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:45:55.953676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:55.953648 5946 patch_prober.go:29] interesting pod/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 16:45:55.953804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:55.953719 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 16:45:55.972849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:55.972825 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:56.129955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:56.129872 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:45:56.273329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:56.273293 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:56.274587 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:56.274564 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4"] Nov 03 16:45:56.278810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:56.278787 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2537c1-tm7p4"] Nov 03 16:45:56.666181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:56.666148 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:56.952633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:56.952551 5946 patch_prober.go:29] interesting pod/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 16:45:56.952633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:56.952598 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 16:45:57.952261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:57.952190 5946 patch_prober.go:29] interesting pod/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 16:45:57.952446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:57.952288 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 16:45:58.005752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:58.005722 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d391ad-12ba-4e68-baec-b5b212f9723d" path="/var/lib/kubelet/pods/85d391ad-12ba-4e68-baec-b5b212f9723d/volumes" Nov 03 16:45:58.951644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:58.951612 5946 patch_prober.go:29] interesting pod/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 16:45:58.951830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:58.951666 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 16:45:59.951838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:59.951801 5946 patch_prober.go:29] interesting pod/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 16:45:59.952499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:45:59.951857 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 16:46:00.952072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:00.952032 5946 patch_prober.go:29] interesting pod/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 16:46:00.952615 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:00.952097 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 16:46:00.988717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:00.988680 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:01.952072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:01.952030 5946 patch_prober.go:29] interesting pod/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 16:46:01.952635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:01.952093 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 16:46:02.957480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:02.957439 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:46:02.958518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:02.958489 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:46:04.280524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:04.280487 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:04.480371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:04.480027 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:04.687812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:04.687720 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:05.552302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:05.552273 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:05.768664 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:05.768621 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:06.150232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.146651 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9"] Nov 03 16:46:06.150232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.147749 5946 cpu_manager.go:414] "containerMap: removing container" podUID="85d391ad-12ba-4e68-baec-b5b212f9723d" containerName="revoke-keys" Nov 03 16:46:06.150232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.147772 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d391ad-12ba-4e68-baec-b5b212f9723d" containerName="revoke-keys" Nov 03 16:46:06.150232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.147934 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d391ad-12ba-4e68-baec-b5b212f9723d" containerName="revoke-keys" Nov 03 16:46:06.165698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.165670 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.242655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.242620 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9"] Nov 03 16:46:06.260519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.260466 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c626cd-openshift-default-dockercfg-hfv6c\"" Nov 03 16:46:06.284221 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.284024 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.331886 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-token\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.331991 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzbcd\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-kube-api-access-jzbcd\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.332057 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-certs\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.332113 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-envoy\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.332244 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96423ef9-ea37-4e35-ad23-015b1bf7598a-istiod-ca-cert\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.332324 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-credential-socket\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.332356 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-data\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.332394 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-podinfo\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.347924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.332495 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-socket\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434192 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434102 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-envoy\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434247 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96423ef9-ea37-4e35-ad23-015b1bf7598a-istiod-ca-cert\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434316 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-credential-socket\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434342 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-data\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434379 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-podinfo\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434471 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-socket\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434568 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-token\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434640 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzbcd\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-kube-api-access-jzbcd\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434690 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-certs\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.434910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.434748 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-data\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.435087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.435065 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-socket\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.435258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.435242 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-certs\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.435360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.435292 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96423ef9-ea37-4e35-ad23-015b1bf7598a-istiod-ca-cert\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.435440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.435414 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-credential-socket\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.437027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.437003 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-envoy\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.437712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.437682 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-podinfo\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.496737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.496705 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-token\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.496921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.496895 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzbcd\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-kube-api-access-jzbcd\") pod \"e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:06.702564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.702479 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:06.795051 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:06.795011 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:07.243754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:07.243722 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9"] Nov 03 16:46:07.244991 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:46:07.244966 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96423ef9_ea37_4e35_ad23_015b1bf7598a.slice/crio-5fa4f9ac52cd0fedb5e02109ffd07f653d37e97f20e8bfe6c868a1a46a7af39e WatchSource:0}: Error finding container 5fa4f9ac52cd0fedb5e02109ffd07f653d37e97f20e8bfe6c868a1a46a7af39e: Status 404 returned error can't find the container with id 5fa4f9ac52cd0fedb5e02109ffd07f653d37e97f20e8bfe6c868a1a46a7af39e Nov 03 16:46:07.248238 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:07.248173 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:46:07.248327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:07.248284 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:46:07.248368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:07.248337 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:46:07.935291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:07.935247 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" event={"ID":"96423ef9-ea37-4e35-ad23-015b1bf7598a","Type":"ContainerStarted","Data":"23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a"} Nov 03 16:46:07.935721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:07.935331 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" event={"ID":"96423ef9-ea37-4e35-ad23-015b1bf7598a","Type":"ContainerStarted","Data":"5fa4f9ac52cd0fedb5e02109ffd07f653d37e97f20e8bfe6c868a1a46a7af39e"} Nov 03 16:46:08.116252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:08.116184 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podStartSLOduration=2.116168552 podStartE2EDuration="2.116168552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:08.115275486 +0000 UTC m=+2038.236328309" watchObservedRunningTime="2026-11-03 16:46:08.116168552 +0000 UTC m=+2038.237221375" Nov 03 16:46:08.795358 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:08.795325 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:08.801676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:08.801652 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:08.943036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:08.943006 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:08.943918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:08.943897 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:46:10.679616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:10.679584 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:11.622117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:11.622079 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:11.698691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:11.698660 5946 scope.go:122] "RemoveContainer" containerID="68d328db32e31ff19ebc991b5b8eb3b897e6d797c4a3bc9ef347adc19fb9305c" Nov 03 16:46:11.820604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:11.820570 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:12.512829 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.512792 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44"] Nov 03 16:46:12.519126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.519100 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.577478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.577432 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44"] Nov 03 16:46:12.591712 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:46:12.591671 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:46:12.617816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.617784 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.617979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.617894 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.617979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.617968 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6p7c\" (UniqueName: \"kubernetes.io/projected/345e17c0-583a-43dc-93ec-5c62ee66f8f9-kube-api-access-d6p7c\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.624311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.624285 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:46:12.624311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.624311 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:46:12.624311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.624318 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:46:12.624549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.624398 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:46:12.719664 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.719630 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.720160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.719717 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.720160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.719789 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6p7c\" (UniqueName: \"kubernetes.io/projected/345e17c0-583a-43dc-93ec-5c62ee66f8f9-kube-api-access-d6p7c\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.720430 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.720406 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.722308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.722290 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.780772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.780701 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6p7c\" (UniqueName: \"kubernetes.io/projected/345e17c0-583a-43dc-93ec-5c62ee66f8f9-kube-api-access-d6p7c\") pod \"maas-api-revoke-keys-e2e-api-a-cb494a-wtx44\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:12.833201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:12.833170 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:13.221139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:13.221112 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44"] Nov 03 16:46:13.222187 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:46:13.222161 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345e17c0_583a_43dc_93ec_5c62ee66f8f9.slice/crio-91f68f2be97a07933cab9f5072314cd68baa2801111538170b2e4a2674178de4 WatchSource:0}: Error finding container 91f68f2be97a07933cab9f5072314cd68baa2801111538170b2e4a2674178de4: Status 404 returned error can't find the container with id 91f68f2be97a07933cab9f5072314cd68baa2801111538170b2e4a2674178de4 Nov 03 16:46:13.982293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:13.982253 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" event={"ID":"345e17c0-583a-43dc-93ec-5c62ee66f8f9","Type":"ContainerStarted","Data":"e2fe04c17434c3e7f3aee307448ef0442ebda970c51ae43d3755657309a7d973"} Nov 03 16:46:13.982740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:13.982316 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" event={"ID":"345e17c0-583a-43dc-93ec-5c62ee66f8f9","Type":"ContainerStarted","Data":"91f68f2be97a07933cab9f5072314cd68baa2801111538170b2e4a2674178de4"} Nov 03 16:46:14.108298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:14.108246 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" podStartSLOduration=2.108232005 podStartE2EDuration="2.108232005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:14.104686769 +0000 UTC m=+2044.225739594" watchObservedRunningTime="2026-11-03 16:46:14.108232005 +0000 UTC m=+2044.229284870" Nov 03 16:46:14.990729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:14.990692 5946 generic.go:356] "Generic (PLEG): container finished" podID="345e17c0-583a-43dc-93ec-5c62ee66f8f9" containerID="e2fe04c17434c3e7f3aee307448ef0442ebda970c51ae43d3755657309a7d973" exitCode=0 Nov 03 16:46:14.991186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:14.990775 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" event={"ID":"345e17c0-583a-43dc-93ec-5c62ee66f8f9","Type":"ContainerDied","Data":"e2fe04c17434c3e7f3aee307448ef0442ebda970c51ae43d3755657309a7d973"} Nov 03 16:46:15.265312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.264970 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:15.492442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.492390 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:15.862330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.862292 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm"] Nov 03 16:46:15.869937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.869907 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm"] Nov 03 16:46:15.870301 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.870282 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:15.921630 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.921601 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9cl\" (UniqueName: \"kubernetes.io/projected/f8cf0120-b157-4ffc-82be-ff5bc16776d4-kube-api-access-js9cl\") pod \"payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:15.922074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.922042 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cf0120-b157-4ffc-82be-ff5bc16776d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:15.927010 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.926978 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c626cd\"" Nov 03 16:46:15.927135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.927117 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c626cd-dockercfg-l6zd2\"" Nov 03 16:46:15.981822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:15.981783 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:16.023262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.023225 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9cl\" (UniqueName: \"kubernetes.io/projected/f8cf0120-b157-4ffc-82be-ff5bc16776d4-kube-api-access-js9cl\") pod \"payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:16.023745 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.023722 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cf0120-b157-4ffc-82be-ff5bc16776d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:16.024535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.024510 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cf0120-b157-4ffc-82be-ff5bc16776d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:16.095755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.093328 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9cl\" (UniqueName: \"kubernetes.io/projected/f8cf0120-b157-4ffc-82be-ff5bc16776d4-kube-api-access-js9cl\") pod \"payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:16.182449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.182419 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:16.213716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.213676 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:16.226028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.225996 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345e17c0-583a-43dc-93ec-5c62ee66f8f9-kube-api-access-d6p7c\" (UniqueName: \"kubernetes.io/projected/345e17c0-583a-43dc-93ec-5c62ee66f8f9-kube-api-access-d6p7c\") pod \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " Nov 03 16:46:16.226390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.226227 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-xks-ca\") pod \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " Nov 03 16:46:16.226390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.226378 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-openshift-ca\") pod \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\" (UID: \"345e17c0-583a-43dc-93ec-5c62ee66f8f9\") " Nov 03 16:46:16.226989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.226958 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-openshift-ca" pod "345e17c0-583a-43dc-93ec-5c62ee66f8f9" (UID: "345e17c0-583a-43dc-93ec-5c62ee66f8f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:16.227326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.227296 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:16.228568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.228544 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345e17c0-583a-43dc-93ec-5c62ee66f8f9-kube-api-access-d6p7c" pod "345e17c0-583a-43dc-93ec-5c62ee66f8f9" (UID: "345e17c0-583a-43dc-93ec-5c62ee66f8f9"). InnerVolumeSpecName "kube-api-access-d6p7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:16.229612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.229549 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-xks-ca" pod "345e17c0-583a-43dc-93ec-5c62ee66f8f9" (UID: "345e17c0-583a-43dc-93ec-5c62ee66f8f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:46:16.328838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.328805 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345e17c0-583a-43dc-93ec-5c62ee66f8f9-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:16.328838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.328834 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6p7c\" (UniqueName: \"kubernetes.io/projected/345e17c0-583a-43dc-93ec-5c62ee66f8f9-kube-api-access-d6p7c\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:16.603093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:16.603062 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm"] Nov 03 16:46:16.603553 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:46:16.603524 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cf0120_b157_4ffc_82be_ff5bc16776d4.slice/crio-5afd7512528d060244b0297d1d5aa448fa58786853bf2dab749b085a473c68f1 WatchSource:0}: Error finding container 5afd7512528d060244b0297d1d5aa448fa58786853bf2dab749b085a473c68f1: Status 404 returned error can't find the container with id 5afd7512528d060244b0297d1d5aa448fa58786853bf2dab749b085a473c68f1 Nov 03 16:46:17.016253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:17.016199 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" event={"ID":"345e17c0-583a-43dc-93ec-5c62ee66f8f9","Type":"ContainerDied","Data":"91f68f2be97a07933cab9f5072314cd68baa2801111538170b2e4a2674178de4"} Nov 03 16:46:17.016453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:17.016273 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91f68f2be97a07933cab9f5072314cd68baa2801111538170b2e4a2674178de4" Nov 03 16:46:17.016453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:17.016225 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44" Nov 03 16:46:17.018021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:17.017993 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" event={"ID":"f8cf0120-b157-4ffc-82be-ff5bc16776d4","Type":"ContainerStarted","Data":"a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4"} Nov 03 16:46:17.018146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:17.018067 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" event={"ID":"f8cf0120-b157-4ffc-82be-ff5bc16776d4","Type":"ContainerStarted","Data":"5afd7512528d060244b0297d1d5aa448fa58786853bf2dab749b085a473c68f1"} Nov 03 16:46:17.018273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:17.018251 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:17.175418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:17.175364 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" podStartSLOduration=2.175348909 podStartE2EDuration="2.175348909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:17.167880073 +0000 UTC m=+2047.288932921" watchObservedRunningTime="2026-11-03 16:46:17.175348909 +0000 UTC m=+2047.296401739" Nov 03 16:46:21.181942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:21.181908 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:21.526021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:21.525965 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:21.574017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:21.573989 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb"] Nov 03 16:46:21.574197 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:21.574178 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" containerID="cri-o://b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22" gracePeriod=30 Nov 03 16:46:21.773175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:21.773141 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:22.925657 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:22.925619 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj"] Nov 03 16:46:22.926149 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:22.925839 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" containerID="cri-o://cedd638919e4873c0ba040bd016e2e5d2cd5ac0720ea3949d480265ebb0cdec5" gracePeriod=30 Nov 03 16:46:23.131650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:23.131619 5946 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:46:23.131822 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:23.131680 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:46:24.459450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:24.459418 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:24.655938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:24.655892 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:26.484752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.484707 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:26.852737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.852714 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:46:26.989812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.989779 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-kube-api-access-k2m8c\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-kube-api-access-k2m8c\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.989998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.989820 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-data\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.989998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.989947 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-socket\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.990136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990037 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-credential-socket\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.990136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990071 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f45c172-3446-4838-92a4-aa1b5a29428f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45c172-3446-4838-92a4-aa1b5a29428f-istiod-ca-cert\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.990308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990144 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-data" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:26.990308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990177 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-certs\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.990308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990266 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-socket" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:26.990308 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990279 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-credential-socket" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:26.990456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990328 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-token\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-token\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.990456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990390 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-certs" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:26.990519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990463 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-envoy\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.990519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990500 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-podinfo\") pod \"8f45c172-3446-4838-92a4-aa1b5a29428f\" (UID: \"8f45c172-3446-4838-92a4-aa1b5a29428f\") " Nov 03 16:46:26.990684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.990509 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f45c172-3446-4838-92a4-aa1b5a29428f-istiod-ca-cert" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:26.991390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.991362 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:26.991390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.991392 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:26.991602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.991408 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:26.991602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.991425 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45c172-3446-4838-92a4-aa1b5a29428f-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:26.991602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.991441 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:26.992386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.992361 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-kube-api-access-k2m8c" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "kube-api-access-k2m8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:26.992622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.992595 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-token" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:26.992716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.992651 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-envoy" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:26.992848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:26.992830 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-podinfo" pod "8f45c172-3446-4838-92a4-aa1b5a29428f" (UID: "8f45c172-3446-4838-92a4-aa1b5a29428f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:27.092306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.092271 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:27.092306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.092298 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:27.092306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.092312 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2m8c\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-kube-api-access-k2m8c\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:27.092535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.092321 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45c172-3446-4838-92a4-aa1b5a29428f-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:27.099527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.099500 5946 generic.go:356] "Generic (PLEG): container finished" podID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerID="b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22" exitCode=0 Nov 03 16:46:27.099663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.099546 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" Nov 03 16:46:27.099663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.099559 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" event={"ID":"8f45c172-3446-4838-92a4-aa1b5a29428f","Type":"ContainerDied","Data":"b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22"} Nov 03 16:46:27.099663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.099598 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb" event={"ID":"8f45c172-3446-4838-92a4-aa1b5a29428f","Type":"ContainerDied","Data":"6b0dc9fe080e7c22dab01f7851a34e1b8e3694673b0d939d17e74dbd4b56659e"} Nov 03 16:46:27.099663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.099620 5946 scope.go:122] "RemoveContainer" containerID="b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22" Nov 03 16:46:27.110880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.110860 5946 scope.go:122] "RemoveContainer" containerID="b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22" Nov 03 16:46:27.111182 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:46:27.111159 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22\": container with ID starting with b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22 not found: ID does not exist" containerID="b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22" Nov 03 16:46:27.111270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.111190 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22"} err="failed to get container status \"b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22\": rpc error: code = NotFound desc = could not find container \"b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22\": container with ID starting with b68433c1ee9ca2fc772be813097e341a971d28aa1b059705c78cbdf14cb3ea22 not found: ID does not exist" Nov 03 16:46:27.274150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.274063 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb"] Nov 03 16:46:27.279627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:27.279597 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-l7ltb"] Nov 03 16:46:28.006325 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.006291 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" path="/var/lib/kubelet/pods/8f45c172-3446-4838-92a4-aa1b5a29428f/volumes" Nov 03 16:46:28.026879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.026850 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:46:28.110753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.110722 5946 generic.go:356] "Generic (PLEG): container finished" podID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerID="cedd638919e4873c0ba040bd016e2e5d2cd5ac0720ea3949d480265ebb0cdec5" exitCode=0 Nov 03 16:46:28.110908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.110840 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" event={"ID":"60d7237e-1293-41c8-b5df-9a9fadfc8c71","Type":"ContainerDied","Data":"cedd638919e4873c0ba040bd016e2e5d2cd5ac0720ea3949d480265ebb0cdec5"} Nov 03 16:46:28.214197 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.214171 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:46:28.307709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307676 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-kube-api-access-rcrns\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-kube-api-access-rcrns\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.307881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307722 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-socket\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.307881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307745 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istiod-ca-cert\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.307881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307801 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-certs\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.307881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307828 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-token\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-token\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.307881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307868 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-credential-socket\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.308174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307926 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-podinfo\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.308174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.307998 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-socket" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:28.308174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308160 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-envoy\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.308377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308221 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-credential-socket" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:28.308377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308291 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-data\") pod \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\" (UID: \"60d7237e-1293-41c8-b5df-9a9fadfc8c71\") " Nov 03 16:46:28.308489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308289 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-certs" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:28.308489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308321 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istiod-ca-cert" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:28.308606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308527 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-data" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:28.308995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308967 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.308995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.308997 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.309239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.309007 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.309239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.309016 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.309239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.309024 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.310715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.310682 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-token" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:28.310839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.310765 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-podinfo" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:28.310839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.310815 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-kube-api-access-rcrns" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "kube-api-access-rcrns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:28.311019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.310993 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-envoy" pod "60d7237e-1293-41c8-b5df-9a9fadfc8c71" (UID: "60d7237e-1293-41c8-b5df-9a9fadfc8c71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:28.411012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.410960 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.411012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.410995 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.411012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.411006 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrns\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-kube-api-access-rcrns\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:28.411012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:28.411015 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d7237e-1293-41c8-b5df-9a9fadfc8c71-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:46:29.121331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:29.121296 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" Nov 03 16:46:29.121749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:29.121292 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj" event={"ID":"60d7237e-1293-41c8-b5df-9a9fadfc8c71","Type":"ContainerDied","Data":"e5daa8dab363a062ddd5d4633217c75f3bb5abb55952f1e92250b7be9ea95988"} Nov 03 16:46:29.121749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:29.121441 5946 scope.go:122] "RemoveContainer" containerID="cedd638919e4873c0ba040bd016e2e5d2cd5ac0720ea3949d480265ebb0cdec5" Nov 03 16:46:29.307523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:29.307491 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj"] Nov 03 16:46:29.359289 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:29.359245 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-xtfmj"] Nov 03 16:46:30.005757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:30.005726 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" path="/var/lib/kubelet/pods/60d7237e-1293-41c8-b5df-9a9fadfc8c71/volumes" Nov 03 16:46:31.391672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:31.391616 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:34.595888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:34.595854 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:46:36.161036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:36.161003 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:36.395730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:36.395698 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:41.407134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:41.407097 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:42.600026 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:46:42.599987 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:46:44.311409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:44.311360 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:46:44.311910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:44.311430 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:46:45.242811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:45.242783 5946 generic.go:356] "Generic (PLEG): container finished" podID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerID="e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730" exitCode=0 Nov 03 16:46:45.242995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:45.242858 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" event={"ID":"d9f08a11-6683-4b7b-a297-7a75aa981220","Type":"ContainerDied","Data":"e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730"} Nov 03 16:46:45.243395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:45.243381 5946 scope.go:122] "RemoveContainer" containerID="e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730" Nov 03 16:46:45.245270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:45.245242 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:46:45.245393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:45.245329 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:46:45.245393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:45.245378 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:46:46.183400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:46.175008 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:46.253053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:46.253002 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" event={"ID":"d9f08a11-6683-4b7b-a297-7a75aa981220","Type":"ContainerStarted","Data":"653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd"} Nov 03 16:46:46.412567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:46.412535 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:46:46.412753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:46.412723 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 16:46:46.412803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:46.412769 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 16:46:47.331236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:47.330795 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:47.412057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:47.412027 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 16:46:47.412246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:47.412081 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 16:46:48.412035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:48.412000 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 16:46:48.412491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:48.412074 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 16:46:49.412635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:49.412600 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 16:46:49.413036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:49.412674 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 16:46:50.412899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:50.412859 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 16:46:50.413461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:50.412919 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 16:46:50.997521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:50.997487 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:51.412645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:51.412558 5946 patch_prober.go:29] interesting pod/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 16:46:51.412645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:51.412617 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 16:46:52.363707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:52.363678 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:52.416155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:52.416126 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:46:52.416524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:52.416505 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:46:52.417198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:52.417180 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:46:55.760521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:55.760484 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:56.850117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850078 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt"] Nov 03 16:46:56.850722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850706 5946 cpu_manager.go:414] "containerMap: removing container" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" Nov 03 16:46:56.850772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850724 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" Nov 03 16:46:56.850772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850734 5946 cpu_manager.go:414] "containerMap: removing container" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" Nov 03 16:46:56.850772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850739 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" Nov 03 16:46:56.850772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850756 5946 cpu_manager.go:414] "containerMap: removing container" podUID="345e17c0-583a-43dc-93ec-5c62ee66f8f9" containerName="revoke-keys" Nov 03 16:46:56.850772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850761 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345e17c0-583a-43dc-93ec-5c62ee66f8f9" containerName="revoke-keys" Nov 03 16:46:56.850948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850866 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f45c172-3446-4838-92a4-aa1b5a29428f" containerName="istio-proxy" Nov 03 16:46:56.850948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850875 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d7237e-1293-41c8-b5df-9a9fadfc8c71" containerName="istio-proxy" Nov 03 16:46:56.850948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.850883 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="345e17c0-583a-43dc-93ec-5c62ee66f8f9" containerName="revoke-keys" Nov 03 16:46:56.854834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.854818 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:56.915572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.915540 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt"] Nov 03 16:46:56.961102 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.961074 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:46:56.961310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.961154 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:46:56.961310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.961171 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:46:56.961310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.961176 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:46:56.975428 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.975398 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:56.975579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.975542 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2hr\" (UniqueName: \"kubernetes.io/projected/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-kube-api-access-2k2hr\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:56.975701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:56.975684 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.077042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.077003 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.077277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.077103 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.077277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.077198 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2hr\" (UniqueName: \"kubernetes.io/projected/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-kube-api-access-2k2hr\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.088037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.078124 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.090310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.090283 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.137549 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.137489 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2hr\" (UniqueName: \"kubernetes.io/projected/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-kube-api-access-2k2hr\") pod \"maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.169476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.169446 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:46:57.555510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:57.555476 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt"] Nov 03 16:46:57.557584 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:46:57.557554 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb63bf45_9ace_41ed_a1d9_48b67b440ba8.slice/crio-d419afc959e2b36e58d05c9a3adf40d6635a7af5296386fd1328a0a235626368 WatchSource:0}: Error finding container d419afc959e2b36e58d05c9a3adf40d6635a7af5296386fd1328a0a235626368: Status 404 returned error can't find the container with id d419afc959e2b36e58d05c9a3adf40d6635a7af5296386fd1328a0a235626368 Nov 03 16:46:58.343605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.343573 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" event={"ID":"fb63bf45-9ace-41ed-a1d9-48b67b440ba8","Type":"ContainerStarted","Data":"8ee0f1ab6ea3e80213c1f3741a76b6361b66e985868f855783fe0787b335ae28"} Nov 03 16:46:58.343998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.343623 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" event={"ID":"fb63bf45-9ace-41ed-a1d9-48b67b440ba8","Type":"ContainerStarted","Data":"d419afc959e2b36e58d05c9a3adf40d6635a7af5296386fd1328a0a235626368"} Nov 03 16:46:58.464411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.464365 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" podStartSLOduration=2.464350411 podStartE2EDuration="2.464350411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:58.463473919 +0000 UTC m=+2088.584526742" watchObservedRunningTime="2026-11-03 16:46:58.464350411 +0000 UTC m=+2088.585403278" Nov 03 16:46:58.493716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.493685 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62"] Nov 03 16:46:58.497924 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.497907 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.581266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.581236 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62"] Nov 03 16:46:58.598465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.598392 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.598659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.598632 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.598780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.598763 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglrm\" (UniqueName: \"kubernetes.io/projected/f8459670-bde2-4336-a992-89ce6c17068f-kube-api-access-nglrm\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.700538 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.700504 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.700738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.700649 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.700738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.700721 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglrm\" (UniqueName: \"kubernetes.io/projected/f8459670-bde2-4336-a992-89ce6c17068f-kube-api-access-nglrm\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.701361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.701334 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.703066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.703046 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.761579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.761541 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglrm\" (UniqueName: \"kubernetes.io/projected/f8459670-bde2-4336-a992-89ce6c17068f-kube-api-access-nglrm\") pod \"maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:58.811387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:58.811354 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:46:59.196946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:59.196919 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62"] Nov 03 16:46:59.197709 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:46:59.197687 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8459670_bde2_4336_a992_89ce6c17068f.slice/crio-6a1d5a5d53ec078827ff785c57ab6cdd7b73fc03ad2b9bac51bff7868aa3eb69 WatchSource:0}: Error finding container 6a1d5a5d53ec078827ff785c57ab6cdd7b73fc03ad2b9bac51bff7868aa3eb69: Status 404 returned error can't find the container with id 6a1d5a5d53ec078827ff785c57ab6cdd7b73fc03ad2b9bac51bff7868aa3eb69 Nov 03 16:46:59.352385 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:59.352346 5946 generic.go:356] "Generic (PLEG): container finished" podID="fb63bf45-9ace-41ed-a1d9-48b67b440ba8" containerID="8ee0f1ab6ea3e80213c1f3741a76b6361b66e985868f855783fe0787b335ae28" exitCode=0 Nov 03 16:46:59.352838 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:59.352388 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" event={"ID":"fb63bf45-9ace-41ed-a1d9-48b67b440ba8","Type":"ContainerDied","Data":"8ee0f1ab6ea3e80213c1f3741a76b6361b66e985868f855783fe0787b335ae28"} Nov 03 16:46:59.353816 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:46:59.353790 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" event={"ID":"f8459670-bde2-4336-a992-89ce6c17068f","Type":"ContainerStarted","Data":"6a1d5a5d53ec078827ff785c57ab6cdd7b73fc03ad2b9bac51bff7868aa3eb69"} Nov 03 16:47:00.162612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.162582 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:00.364403 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.364368 5946 generic.go:356] "Generic (PLEG): container finished" podID="f8459670-bde2-4336-a992-89ce6c17068f" containerID="d1a63fe927de5a6c844516126f4f43f0fe153389ec48b5e446617481b153bff9" exitCode=0 Nov 03 16:47:00.364910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.364463 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" event={"ID":"f8459670-bde2-4336-a992-89ce6c17068f","Type":"ContainerDied","Data":"d1a63fe927de5a6c844516126f4f43f0fe153389ec48b5e446617481b153bff9"} Nov 03 16:47:00.735693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.735656 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:47:00.867271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.867153 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-openshift-ca\") pod \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " Nov 03 16:47:00.867271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.867262 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-xks-ca\") pod \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " Nov 03 16:47:00.867795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.867763 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-kube-api-access-2k2hr\" (UniqueName: \"kubernetes.io/projected/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-kube-api-access-2k2hr\") pod \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\" (UID: \"fb63bf45-9ace-41ed-a1d9-48b67b440ba8\") " Nov 03 16:47:00.868032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.867768 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-openshift-ca" pod "fb63bf45-9ace-41ed-a1d9-48b67b440ba8" (UID: "fb63bf45-9ace-41ed-a1d9-48b67b440ba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:00.868485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.868457 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:00.870382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.870270 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-xks-ca" pod "fb63bf45-9ace-41ed-a1d9-48b67b440ba8" (UID: "fb63bf45-9ace-41ed-a1d9-48b67b440ba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:00.871007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.870982 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-kube-api-access-2k2hr" pod "fb63bf45-9ace-41ed-a1d9-48b67b440ba8" (UID: "fb63bf45-9ace-41ed-a1d9-48b67b440ba8"). InnerVolumeSpecName "kube-api-access-2k2hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:00.970015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.969982 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:00.970015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:00.970011 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2hr\" (UniqueName: \"kubernetes.io/projected/fb63bf45-9ace-41ed-a1d9-48b67b440ba8-kube-api-access-2k2hr\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:01.372865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.372825 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" event={"ID":"fb63bf45-9ace-41ed-a1d9-48b67b440ba8","Type":"ContainerDied","Data":"d419afc959e2b36e58d05c9a3adf40d6635a7af5296386fd1328a0a235626368"} Nov 03 16:47:01.373399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.372890 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d419afc959e2b36e58d05c9a3adf40d6635a7af5296386fd1328a0a235626368" Nov 03 16:47:01.373399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.372847 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt" Nov 03 16:47:01.547386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.547362 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:47:01.686338 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.686257 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-xks-ca\") pod \"f8459670-bde2-4336-a992-89ce6c17068f\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " Nov 03 16:47:01.686592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.686570 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8459670-bde2-4336-a992-89ce6c17068f-kube-api-access-nglrm\" (UniqueName: \"kubernetes.io/projected/f8459670-bde2-4336-a992-89ce6c17068f-kube-api-access-nglrm\") pod \"f8459670-bde2-4336-a992-89ce6c17068f\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " Nov 03 16:47:01.686671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.686655 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-openshift-ca\") pod \"f8459670-bde2-4336-a992-89ce6c17068f\" (UID: \"f8459670-bde2-4336-a992-89ce6c17068f\") " Nov 03 16:47:01.687100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.687067 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-openshift-ca" pod "f8459670-bde2-4336-a992-89ce6c17068f" (UID: "f8459670-bde2-4336-a992-89ce6c17068f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:01.687260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.687164 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:01.689157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.689129 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-xks-ca" pod "f8459670-bde2-4336-a992-89ce6c17068f" (UID: "f8459670-bde2-4336-a992-89ce6c17068f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:01.689288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.689242 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8459670-bde2-4336-a992-89ce6c17068f-kube-api-access-nglrm" pod "f8459670-bde2-4336-a992-89ce6c17068f" (UID: "f8459670-bde2-4336-a992-89ce6c17068f"). InnerVolumeSpecName "kube-api-access-nglrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:01.788082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.788055 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglrm\" (UniqueName: \"kubernetes.io/projected/f8459670-bde2-4336-a992-89ce6c17068f-kube-api-access-nglrm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:01.788082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:01.788080 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8459670-bde2-4336-a992-89ce6c17068f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:02.001127 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:02.001060 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:02.174618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:02.174583 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76"] Nov 03 16:47:02.179096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:02.179071 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9635db0d-v6r76"] Nov 03 16:47:02.220063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:02.220024 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:02.381584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:02.381511 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" Nov 03 16:47:02.381584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:02.381520 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62" event={"ID":"f8459670-bde2-4336-a992-89ce6c17068f","Type":"ContainerDied","Data":"6a1d5a5d53ec078827ff785c57ab6cdd7b73fc03ad2b9bac51bff7868aa3eb69"} Nov 03 16:47:02.381584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:02.381561 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a1d5a5d53ec078827ff785c57ab6cdd7b73fc03ad2b9bac51bff7868aa3eb69" Nov 03 16:47:04.006815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.006782 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc3763a-23ed-4817-9d66-ca1bfc3cd192" path="/var/lib/kubelet/pods/8cc3763a-23ed-4817-9d66-ca1bfc3cd192/volumes" Nov 03 16:47:04.155977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.155937 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82"] Nov 03 16:47:04.156580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.156565 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f8459670-bde2-4336-a992-89ce6c17068f" containerName="revoke-keys" Nov 03 16:47:04.156639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.156582 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8459670-bde2-4336-a992-89ce6c17068f" containerName="revoke-keys" Nov 03 16:47:04.156639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.156606 5946 cpu_manager.go:414] "containerMap: removing container" podUID="fb63bf45-9ace-41ed-a1d9-48b67b440ba8" containerName="revoke-keys" Nov 03 16:47:04.156639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.156611 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb63bf45-9ace-41ed-a1d9-48b67b440ba8" containerName="revoke-keys" Nov 03 16:47:04.156734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.156727 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8459670-bde2-4336-a992-89ce6c17068f" containerName="revoke-keys" Nov 03 16:47:04.156768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.156739 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb63bf45-9ace-41ed-a1d9-48b67b440ba8" containerName="revoke-keys" Nov 03 16:47:04.160491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.160475 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.219281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.219251 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82"] Nov 03 16:47:04.224610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.224581 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktvh\" (UniqueName: \"kubernetes.io/projected/82783799-5fb1-44dc-9b0c-bb36a4de9c46-kube-api-access-9ktvh\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.224781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.224732 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.224849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.224832 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.266233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.266135 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:47:04.266233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.266150 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:47:04.266233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.266150 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:47:04.266457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.266331 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:47:04.326103 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.326066 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.326310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.326179 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktvh\" (UniqueName: \"kubernetes.io/projected/82783799-5fb1-44dc-9b0c-bb36a4de9c46-kube-api-access-9ktvh\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.326392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.326373 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.327006 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.326987 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.328669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.328645 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.387152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.387114 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktvh\" (UniqueName: \"kubernetes.io/projected/82783799-5fb1-44dc-9b0c-bb36a4de9c46-kube-api-access-9ktvh\") pod \"maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.474670 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.474639 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:04.867858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:04.867829 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82"] Nov 03 16:47:04.868696 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:47:04.868666 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82783799_5fb1_44dc_9b0c_bb36a4de9c46.slice/crio-617f9b971d5e2d697f720e7efc7c7544ac170b6387f0d13729ab4a329c511d03 WatchSource:0}: Error finding container 617f9b971d5e2d697f720e7efc7c7544ac170b6387f0d13729ab4a329c511d03: Status 404 returned error can't find the container with id 617f9b971d5e2d697f720e7efc7c7544ac170b6387f0d13729ab4a329c511d03 Nov 03 16:47:05.406324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:05.406246 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" event={"ID":"82783799-5fb1-44dc-9b0c-bb36a4de9c46","Type":"ContainerStarted","Data":"c4be8dd63cfeee3a31606c4bcf85b2e933a4bdbd470288e304a84f19c81bad38"} Nov 03 16:47:05.406324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:05.406294 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" event={"ID":"82783799-5fb1-44dc-9b0c-bb36a4de9c46","Type":"ContainerStarted","Data":"617f9b971d5e2d697f720e7efc7c7544ac170b6387f0d13729ab4a329c511d03"} Nov 03 16:47:05.527108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:05.527054 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" podStartSLOduration=1.52704002 podStartE2EDuration="1.52704002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:05.526525397 +0000 UTC m=+2095.647578221" watchObservedRunningTime="2026-11-03 16:47:05.52704002 +0000 UTC m=+2095.648092842" Nov 03 16:47:06.415933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:06.415896 5946 generic.go:356] "Generic (PLEG): container finished" podID="82783799-5fb1-44dc-9b0c-bb36a4de9c46" containerID="c4be8dd63cfeee3a31606c4bcf85b2e933a4bdbd470288e304a84f19c81bad38" exitCode=0 Nov 03 16:47:06.416452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:06.415980 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" event={"ID":"82783799-5fb1-44dc-9b0c-bb36a4de9c46","Type":"ContainerDied","Data":"c4be8dd63cfeee3a31606c4bcf85b2e933a4bdbd470288e304a84f19c81bad38"} Nov 03 16:47:07.484700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.484653 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:07.608904 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.608878 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:07.675039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.675008 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-openshift-ca\") pod \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " Nov 03 16:47:07.675245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.675125 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82783799-5fb1-44dc-9b0c-bb36a4de9c46-kube-api-access-9ktvh\" (UniqueName: \"kubernetes.io/projected/82783799-5fb1-44dc-9b0c-bb36a4de9c46-kube-api-access-9ktvh\") pod \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " Nov 03 16:47:07.675320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.675265 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-xks-ca\") pod \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\" (UID: \"82783799-5fb1-44dc-9b0c-bb36a4de9c46\") " Nov 03 16:47:07.675462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.675438 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-openshift-ca" pod "82783799-5fb1-44dc-9b0c-bb36a4de9c46" (UID: "82783799-5fb1-44dc-9b0c-bb36a4de9c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:07.675707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.675683 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:07.677548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.677482 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82783799-5fb1-44dc-9b0c-bb36a4de9c46-kube-api-access-9ktvh" pod "82783799-5fb1-44dc-9b0c-bb36a4de9c46" (UID: "82783799-5fb1-44dc-9b0c-bb36a4de9c46"). InnerVolumeSpecName "kube-api-access-9ktvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:07.677548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.677506 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-xks-ca" pod "82783799-5fb1-44dc-9b0c-bb36a4de9c46" (UID: "82783799-5fb1-44dc-9b0c-bb36a4de9c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:07.777228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.777178 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ktvh\" (UniqueName: \"kubernetes.io/projected/82783799-5fb1-44dc-9b0c-bb36a4de9c46-kube-api-access-9ktvh\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:07.777228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:07.777227 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82783799-5fb1-44dc-9b0c-bb36a4de9c46-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:08.434339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:08.434248 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" event={"ID":"82783799-5fb1-44dc-9b0c-bb36a4de9c46","Type":"ContainerDied","Data":"617f9b971d5e2d697f720e7efc7c7544ac170b6387f0d13729ab4a329c511d03"} Nov 03 16:47:08.434339 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:08.434305 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="617f9b971d5e2d697f720e7efc7c7544ac170b6387f0d13729ab4a329c511d03" Nov 03 16:47:08.434563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:08.434370 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82" Nov 03 16:47:09.769160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:09.769116 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:09.966180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:09.966133 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:10.356078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:10.356049 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:47:10.361136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:10.361118 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:47:11.921898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:11.921856 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:11.931097 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:11.931062 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r"] Nov 03 16:47:12.357436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:12.357392 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:12.554800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:12.554749 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:12.589253 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:12.589200 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:47:13.863240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:13.863153 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:16.932382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:16.932338 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" podUID="9135cfdb-7c1c-4a8b-833b-db32a0261825" containerName="payload-processing" containerID="cri-o://95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced" gracePeriod=25 Nov 03 16:47:17.048555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.048524 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:17.236915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.236467 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:17.268716 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.268691 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:47:17.322909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.322872 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9135cfdb-7c1c-4a8b-833b-db32a0261825-kube-api-access-4wkvt\" (UniqueName: \"kubernetes.io/projected/9135cfdb-7c1c-4a8b-833b-db32a0261825-kube-api-access-4wkvt\") pod \"9135cfdb-7c1c-4a8b-833b-db32a0261825\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " Nov 03 16:47:17.323087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.322931 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9135cfdb-7c1c-4a8b-833b-db32a0261825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9135cfdb-7c1c-4a8b-833b-db32a0261825-plugins-config-volume\") pod \"9135cfdb-7c1c-4a8b-833b-db32a0261825\" (UID: \"9135cfdb-7c1c-4a8b-833b-db32a0261825\") " Nov 03 16:47:17.323372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.323346 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9135cfdb-7c1c-4a8b-833b-db32a0261825-plugins-config-volume" pod "9135cfdb-7c1c-4a8b-833b-db32a0261825" (UID: "9135cfdb-7c1c-4a8b-833b-db32a0261825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:17.323460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.323446 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9135cfdb-7c1c-4a8b-833b-db32a0261825-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:17.325189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.325171 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9135cfdb-7c1c-4a8b-833b-db32a0261825-kube-api-access-4wkvt" pod "9135cfdb-7c1c-4a8b-833b-db32a0261825" (UID: "9135cfdb-7c1c-4a8b-833b-db32a0261825"). InnerVolumeSpecName "kube-api-access-4wkvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:17.424616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.424580 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wkvt\" (UniqueName: \"kubernetes.io/projected/9135cfdb-7c1c-4a8b-833b-db32a0261825-kube-api-access-4wkvt\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:17.507151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.507052 5946 generic.go:356] "Generic (PLEG): container finished" podID="9135cfdb-7c1c-4a8b-833b-db32a0261825" containerID="95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced" exitCode=0 Nov 03 16:47:17.507151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.507101 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" Nov 03 16:47:17.507151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.507131 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" event={"ID":"9135cfdb-7c1c-4a8b-833b-db32a0261825","Type":"ContainerDied","Data":"95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced"} Nov 03 16:47:17.507486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.507188 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r" event={"ID":"9135cfdb-7c1c-4a8b-833b-db32a0261825","Type":"ContainerDied","Data":"84285c5c34b904d87a4c2c55c149b8bc0d1c9df1b1a5593d783f3a21b8994028"} Nov 03 16:47:17.507486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.507251 5946 scope.go:122] "RemoveContainer" containerID="95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced" Nov 03 16:47:17.518647 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.518630 5946 scope.go:122] "RemoveContainer" containerID="95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced" Nov 03 16:47:17.518928 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:17.518909 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced\": container with ID starting with 95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced not found: ID does not exist" containerID="95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced" Nov 03 16:47:17.518998 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.518937 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced"} err="failed to get container status \"95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced\": rpc error: code = NotFound desc = could not find container \"95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced\": container with ID starting with 95ef9e55bb86e5628fc1bfed8c79755b3d88b3ec258d43931d0775310ff59ced not found: ID does not exist" Nov 03 16:47:17.681078 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.681047 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r"] Nov 03 16:47:17.685377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:17.685348 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6822cb-796f7c44b7-chn4r"] Nov 03 16:47:18.005764 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:18.005736 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9135cfdb-7c1c-4a8b-833b-db32a0261825" path="/var/lib/kubelet/pods/9135cfdb-7c1c-4a8b-833b-db32a0261825/volumes" Nov 03 16:47:18.226691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:18.226657 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:18.429682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:18.429589 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:20.179811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:20.179775 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:21.653131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:21.653100 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:21.978314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:21.978195 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:22.159356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:22.159323 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62"] Nov 03 16:47:22.210125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:22.210087 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6822cb-8gc62"] Nov 03 16:47:22.623016 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:22.622984 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:24.006124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:24.006095 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8459670-bde2-4336-a992-89ce6c17068f" path="/var/lib/kubelet/pods/f8459670-bde2-4336-a992-89ce6c17068f/volumes" Nov 03 16:47:25.391005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:25.390961 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:25.582068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:25.582025 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:25.778964 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:25.778928 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:30.145837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:30.145779 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:30.668850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:30.668817 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:30.858860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:30.858823 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:31.147639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:31.147596 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v"] Nov 03 16:47:31.148175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:31.147864 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" containerID="cri-o://653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd" gracePeriod=30 Nov 03 16:47:31.248741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:31.248709 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:32.282977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:32.282924 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:33.079905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.079873 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:33.626653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.626612 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k"] Nov 03 16:47:33.628019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.627987 5946 cpu_manager.go:414] "containerMap: removing container" podUID="82783799-5fb1-44dc-9b0c-bb36a4de9c46" containerName="revoke-keys" Nov 03 16:47:33.628019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.628018 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82783799-5fb1-44dc-9b0c-bb36a4de9c46" containerName="revoke-keys" Nov 03 16:47:33.628261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.628041 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9135cfdb-7c1c-4a8b-833b-db32a0261825" containerName="payload-processing" Nov 03 16:47:33.628261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.628049 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9135cfdb-7c1c-4a8b-833b-db32a0261825" containerName="payload-processing" Nov 03 16:47:33.628495 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.628406 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="82783799-5fb1-44dc-9b0c-bb36a4de9c46" containerName="revoke-keys" Nov 03 16:47:33.628495 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.628435 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9135cfdb-7c1c-4a8b-833b-db32a0261825" containerName="payload-processing" Nov 03 16:47:33.633601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.633581 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.689505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.689474 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:47:33.692191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.692158 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k"] Nov 03 16:47:33.711444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.711418 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.711612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.711570 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.711712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.711690 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdf7\" (UniqueName: \"kubernetes.io/projected/411c49f0-8316-415a-af3e-b97b8a250e12-kube-api-access-tmdf7\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.738478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.738444 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:47:33.740287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.740261 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:47:33.740409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.740365 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:47:33.813259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.813219 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.813454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.813288 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdf7\" (UniqueName: \"kubernetes.io/projected/411c49f0-8316-415a-af3e-b97b8a250e12-kube-api-access-tmdf7\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.813454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.813422 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.813904 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.813877 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.815952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.815933 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.873946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.873914 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdf7\" (UniqueName: \"kubernetes.io/projected/411c49f0-8316-415a-af3e-b97b8a250e12-kube-api-access-tmdf7\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:33.948266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:33.948166 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:34.333458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:34.333429 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k"] Nov 03 16:47:34.333991 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:47:34.333965 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411c49f0_8316_415a_af3e_b97b8a250e12.slice/crio-980ff1dc887ae03c741e1af5231b9b9bb3f2c2464fcef9cfe5b41860200de81c WatchSource:0}: Error finding container 980ff1dc887ae03c741e1af5231b9b9bb3f2c2464fcef9cfe5b41860200de81c: Status 404 returned error can't find the container with id 980ff1dc887ae03c741e1af5231b9b9bb3f2c2464fcef9cfe5b41860200de81c Nov 03 16:47:34.725302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:34.725266 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" event={"ID":"411c49f0-8316-415a-af3e-b97b8a250e12","Type":"ContainerStarted","Data":"5d478cbd98d35ad08b2a7b5e1ab33735a190c5b6df8d5a61ad06d6a2bc402bd7"} Nov 03 16:47:34.725732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:34.725330 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" event={"ID":"411c49f0-8316-415a-af3e-b97b8a250e12","Type":"ContainerStarted","Data":"980ff1dc887ae03c741e1af5231b9b9bb3f2c2464fcef9cfe5b41860200de81c"} Nov 03 16:47:34.849282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:34.849233 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" podStartSLOduration=1.849219115 podStartE2EDuration="1.849219115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:34.84696385 +0000 UTC m=+2124.968016686" watchObservedRunningTime="2026-11-03 16:47:34.849219115 +0000 UTC m=+2124.970271934" Nov 03 16:47:35.712120 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:35.712020 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:35.738756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:35.738719 5946 generic.go:356] "Generic (PLEG): container finished" podID="411c49f0-8316-415a-af3e-b97b8a250e12" containerID="5d478cbd98d35ad08b2a7b5e1ab33735a190c5b6df8d5a61ad06d6a2bc402bd7" exitCode=0 Nov 03 16:47:35.739342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:35.738800 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" event={"ID":"411c49f0-8316-415a-af3e-b97b8a250e12","Type":"ContainerDied","Data":"5d478cbd98d35ad08b2a7b5e1ab33735a190c5b6df8d5a61ad06d6a2bc402bd7"} Nov 03 16:47:35.967925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:35.967838 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:36.154281 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.154247 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:36.433267 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.433237 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:47:36.468366 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.468340 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:36.553691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553611 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-podinfo\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.553691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553669 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-credential-socket\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.553915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553709 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-socket\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.553915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553790 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-data\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.553915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553839 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-kube-api-access-ft42s\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-kube-api-access-ft42s\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.553915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553880 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f08a11-6683-4b7b-a297-7a75aa981220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f08a11-6683-4b7b-a297-7a75aa981220-istiod-ca-cert\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.553915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553914 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-envoy\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.554189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.553968 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-token\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-token\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.554189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554005 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-certs\") pod \"d9f08a11-6683-4b7b-a297-7a75aa981220\" (UID: \"d9f08a11-6683-4b7b-a297-7a75aa981220\") " Nov 03 16:47:36.554189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554020 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-credential-socket" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:36.554189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554088 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-data" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:36.554461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554311 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-socket" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:36.554461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554346 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-certs" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:36.554573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554492 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f08a11-6683-4b7b-a297-7a75aa981220-istiod-ca-cert" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:36.554908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554863 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.554908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554894 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.554908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554909 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f08a11-6683-4b7b-a297-7a75aa981220-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.555377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554923 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.555377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.554938 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.556372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.556345 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-podinfo" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:47:36.556583 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.556556 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-kube-api-access-ft42s" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "kube-api-access-ft42s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:36.556691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.556653 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-envoy" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:36.556823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.556806 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-token" pod "d9f08a11-6683-4b7b-a297-7a75aa981220" (UID: "d9f08a11-6683-4b7b-a297-7a75aa981220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:36.655847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.655808 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft42s\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-kube-api-access-ft42s\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.655847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.655838 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.655847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.655847 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.655847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.655855 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f08a11-6683-4b7b-a297-7a75aa981220-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:36.748075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.748039 5946 generic.go:356] "Generic (PLEG): container finished" podID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerID="653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd" exitCode=0 Nov 03 16:47:36.748548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.748093 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" Nov 03 16:47:36.748548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.748120 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" event={"ID":"d9f08a11-6683-4b7b-a297-7a75aa981220","Type":"ContainerDied","Data":"653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd"} Nov 03 16:47:36.748548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.748175 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v" event={"ID":"d9f08a11-6683-4b7b-a297-7a75aa981220","Type":"ContainerDied","Data":"059cac0978ce73a0dd7c77aecf302e1c14aacd909f1ec20b512346036f9b37e7"} Nov 03 16:47:36.748548 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.748199 5946 scope.go:122] "RemoveContainer" containerID="653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd" Nov 03 16:47:36.760868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.760848 5946 scope.go:122] "RemoveContainer" containerID="e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730" Nov 03 16:47:36.771678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.771655 5946 scope.go:122] "RemoveContainer" containerID="653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd" Nov 03 16:47:36.772012 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:36.771990 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd\": container with ID starting with 653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd not found: ID does not exist" containerID="653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd" Nov 03 16:47:36.772108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.772021 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd"} err="failed to get container status \"653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd\": rpc error: code = NotFound desc = could not find container \"653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd\": container with ID starting with 653d08fa43a3a96cd377b547e66f6943ecd93dee1ff1a9a778fa815dd0c0c5fd not found: ID does not exist" Nov 03 16:47:36.772108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.772039 5946 scope.go:122] "RemoveContainer" containerID="e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730" Nov 03 16:47:36.772347 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:36.772329 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730\": container with ID starting with e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730 not found: ID does not exist" containerID="e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730" Nov 03 16:47:36.772411 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.772355 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730"} err="failed to get container status \"e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730\": rpc error: code = NotFound desc = could not find container \"e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730\": container with ID starting with e01a100a711b90dd50226ba29a16b3eb00b99563c8b00e2b1587be2478c68730 not found: ID does not exist" Nov 03 16:47:36.883579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.883550 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:36.922311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.922277 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v"] Nov 03 16:47:36.928115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.928090 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6822cb-openshift-default-67cd55cbfc-k8h7v"] Nov 03 16:47:36.958926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.958899 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411c49f0-8316-415a-af3e-b97b8a250e12-kube-api-access-tmdf7\" (UniqueName: \"kubernetes.io/projected/411c49f0-8316-415a-af3e-b97b8a250e12-kube-api-access-tmdf7\") pod \"411c49f0-8316-415a-af3e-b97b8a250e12\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " Nov 03 16:47:36.959083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.958957 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-xks-ca\") pod \"411c49f0-8316-415a-af3e-b97b8a250e12\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " Nov 03 16:47:36.959083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.958981 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-openshift-ca\") pod \"411c49f0-8316-415a-af3e-b97b8a250e12\" (UID: \"411c49f0-8316-415a-af3e-b97b8a250e12\") " Nov 03 16:47:36.959455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.959431 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-openshift-ca" pod "411c49f0-8316-415a-af3e-b97b8a250e12" (UID: "411c49f0-8316-415a-af3e-b97b8a250e12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:36.961167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.961142 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411c49f0-8316-415a-af3e-b97b8a250e12-kube-api-access-tmdf7" pod "411c49f0-8316-415a-af3e-b97b8a250e12" (UID: "411c49f0-8316-415a-af3e-b97b8a250e12"). InnerVolumeSpecName "kube-api-access-tmdf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:36.961283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:36.961268 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-xks-ca" pod "411c49f0-8316-415a-af3e-b97b8a250e12" (UID: "411c49f0-8316-415a-af3e-b97b8a250e12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:37.060659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:37.060578 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdf7\" (UniqueName: \"kubernetes.io/projected/411c49f0-8316-415a-af3e-b97b8a250e12-kube-api-access-tmdf7\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:37.060659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:37.060620 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:37.060659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:37.060639 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411c49f0-8316-415a-af3e-b97b8a250e12-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:37.758453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:37.758416 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" event={"ID":"411c49f0-8316-415a-af3e-b97b8a250e12","Type":"ContainerDied","Data":"980ff1dc887ae03c741e1af5231b9b9bb3f2c2464fcef9cfe5b41860200de81c"} Nov 03 16:47:37.758959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:37.758431 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k" Nov 03 16:47:37.758959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:37.758517 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="980ff1dc887ae03c741e1af5231b9b9bb3f2c2464fcef9cfe5b41860200de81c" Nov 03 16:47:37.974303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:37.974268 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk"] Nov 03 16:47:38.007160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:38.007127 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" path="/var/lib/kubelet/pods/d9f08a11-6683-4b7b-a297-7a75aa981220/volumes" Nov 03 16:47:38.176286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:38.176184 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:39.531042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:39.531006 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:39.728945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:39.728914 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:39.852309 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:39.852226 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687"] Nov 03 16:47:39.852478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:39.852434 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" containerID="cri-o://920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455" gracePeriod=30 Nov 03 16:47:39.921919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:39.921872 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:40.745789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:40.745758 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:40.936457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:40.936422 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:42.585751 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:42.585714 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:47:45.147554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.147525 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:47:45.275139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275102 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-kube-api-access-2f7wx\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-kube-api-access-2f7wx\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275159 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-data\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275191 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-credential-socket\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275259 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-podinfo\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275508 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-data" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:45.275634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275516 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-credential-socket" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:45.275634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275526 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-certs\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275569 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istiod-ca-cert\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275661 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-token\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-token\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275714 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-envoy\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275742 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-socket\") pod \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\" (UID: \"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f\") " Nov 03 16:47:45.275956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275747 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-certs" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:45.275956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.275923 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istiod-ca-cert" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:45.276285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.276130 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-socket" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:45.276418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.276399 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.276488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.276422 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.276488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.276433 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.276488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.276442 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.276488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.276450 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.277753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.277728 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-podinfo" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:47:45.278188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.278166 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-token" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:45.278298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.278269 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-kube-api-access-2f7wx" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "kube-api-access-2f7wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:45.278457 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.278435 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-envoy" pod "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" (UID: "76bdfccd-eb96-409f-9f80-d4cecc7a6a5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:45.377818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.377719 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.377818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.377746 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.377818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.377759 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f7wx\" (UniqueName: \"kubernetes.io/projected/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-kube-api-access-2f7wx\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.377818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.377777 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:45.538837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.538800 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:45.736088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.736030 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:45.865031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.864993 5946 generic.go:356] "Generic (PLEG): container finished" podID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerID="920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455" exitCode=0 Nov 03 16:47:45.865248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.865058 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" Nov 03 16:47:45.865248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.865072 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" event={"ID":"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f","Type":"ContainerDied","Data":"920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455"} Nov 03 16:47:45.865248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.865176 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687" event={"ID":"76bdfccd-eb96-409f-9f80-d4cecc7a6a5f","Type":"ContainerDied","Data":"76fc2d1303ab36afc10f77e2bc6b6283157660d6f8f3dc245553af57ab4079c4"} Nov 03 16:47:45.865248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.865229 5946 scope.go:122] "RemoveContainer" containerID="920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455" Nov 03 16:47:45.877188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.877173 5946 scope.go:122] "RemoveContainer" containerID="bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a" Nov 03 16:47:45.887303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.887287 5946 scope.go:122] "RemoveContainer" containerID="920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455" Nov 03 16:47:45.887559 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:45.887540 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455\": container with ID starting with 920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455 not found: ID does not exist" containerID="920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455" Nov 03 16:47:45.887623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.887565 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455"} err="failed to get container status \"920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455\": rpc error: code = NotFound desc = could not find container \"920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455\": container with ID starting with 920cbfeb819a81feb1ba2513f615e7ed76a295fc4e15ef56127f3619aace2455 not found: ID does not exist" Nov 03 16:47:45.887623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.887582 5946 scope.go:122] "RemoveContainer" containerID="bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a" Nov 03 16:47:45.887821 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:45.887807 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a\": container with ID starting with bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a not found: ID does not exist" containerID="bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a" Nov 03 16:47:45.887870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:45.887824 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a"} err="failed to get container status \"bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a\": rpc error: code = NotFound desc = could not find container \"bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a\": container with ID starting with bca71d6d7c47bc1eacc0ff1924fbbcefe66abe0388b727959f720d9d36653c2a not found: ID does not exist" Nov 03 16:47:46.047091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:46.047011 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687"] Nov 03 16:47:46.097811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:46.097781 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9635db0d-openshift-default-5c876b8797-km687"] Nov 03 16:47:46.228630 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:46.218455 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:48.006490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:48.006448 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" path="/var/lib/kubelet/pods/76bdfccd-eb96-409f-9f80-d4cecc7a6a5f/volumes" Nov 03 16:47:51.522906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:51.522788 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:52.996892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:52.996846 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" containerID="cri-o://47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6" gracePeriod=45 Nov 03 16:47:53.275951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.275925 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:47:53.290077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290040 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-home\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-home\") pod \"40091c43-f73f-4b47-b221-9567bb1670ee\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " Nov 03 16:47:53.290274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290126 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-tmp-dir\") pod \"40091c43-f73f-4b47-b221-9567bb1670ee\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " Nov 03 16:47:53.290274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290252 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40091c43-f73f-4b47-b221-9567bb1670ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/40091c43-f73f-4b47-b221-9567bb1670ee-tls-certs\") pod \"40091c43-f73f-4b47-b221-9567bb1670ee\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " Nov 03 16:47:53.290421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290276 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-dshm\") pod \"40091c43-f73f-4b47-b221-9567bb1670ee\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " Nov 03 16:47:53.290421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290316 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-model-cache\") pod \"40091c43-f73f-4b47-b221-9567bb1670ee\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " Nov 03 16:47:53.290421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290337 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-home" pod "40091c43-f73f-4b47-b221-9567bb1670ee" (UID: "40091c43-f73f-4b47-b221-9567bb1670ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:53.290661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290440 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40091c43-f73f-4b47-b221-9567bb1670ee-kube-api-access-2jkfp\" (UniqueName: \"kubernetes.io/projected/40091c43-f73f-4b47-b221-9567bb1670ee-kube-api-access-2jkfp\") pod \"40091c43-f73f-4b47-b221-9567bb1670ee\" (UID: \"40091c43-f73f-4b47-b221-9567bb1670ee\") " Nov 03 16:47:53.290661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290444 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-tmp-dir" pod "40091c43-f73f-4b47-b221-9567bb1670ee" (UID: "40091c43-f73f-4b47-b221-9567bb1670ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:53.290765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.290658 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-model-cache" pod "40091c43-f73f-4b47-b221-9567bb1670ee" (UID: "40091c43-f73f-4b47-b221-9567bb1670ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:53.291275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.291229 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:53.291275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.291253 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:53.291275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.291269 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:53.292769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.292744 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-dshm" pod "40091c43-f73f-4b47-b221-9567bb1670ee" (UID: "40091c43-f73f-4b47-b221-9567bb1670ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:53.292878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.292838 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40091c43-f73f-4b47-b221-9567bb1670ee-kube-api-access-2jkfp" pod "40091c43-f73f-4b47-b221-9567bb1670ee" (UID: "40091c43-f73f-4b47-b221-9567bb1670ee"). InnerVolumeSpecName "kube-api-access-2jkfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:53.292878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.292838 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40091c43-f73f-4b47-b221-9567bb1670ee-tls-certs" pod "40091c43-f73f-4b47-b221-9567bb1670ee" (UID: "40091c43-f73f-4b47-b221-9567bb1670ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:53.392849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.392812 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jkfp\" (UniqueName: \"kubernetes.io/projected/40091c43-f73f-4b47-b221-9567bb1670ee-kube-api-access-2jkfp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:53.392849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.392845 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091c43-f73f-4b47-b221-9567bb1670ee-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:53.393068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.392859 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091c43-f73f-4b47-b221-9567bb1670ee-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:47:53.923097 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.923062 5946 generic.go:356] "Generic (PLEG): container finished" podID="40091c43-f73f-4b47-b221-9567bb1670ee" containerID="47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6" exitCode=0 Nov 03 16:47:53.923317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.923107 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" Nov 03 16:47:53.923317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.923142 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" event={"ID":"40091c43-f73f-4b47-b221-9567bb1670ee","Type":"ContainerDied","Data":"47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6"} Nov 03 16:47:53.923317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.923179 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk" event={"ID":"40091c43-f73f-4b47-b221-9567bb1670ee","Type":"ContainerDied","Data":"abe95090e88a52891aaa118788e725bdf8e6b4fbd520ced0292c0afa5408fb6e"} Nov 03 16:47:53.923317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.923195 5946 scope.go:122] "RemoveContainer" containerID="47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6" Nov 03 16:47:53.934934 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.934916 5946 scope.go:122] "RemoveContainer" containerID="47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6" Nov 03 16:47:53.935187 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:47:53.935165 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6\": container with ID starting with 47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6 not found: ID does not exist" containerID="47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6" Nov 03 16:47:53.935284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:53.935199 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6"} err="failed to get container status \"47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6\": rpc error: code = NotFound desc = could not find container \"47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6\": container with ID starting with 47ae12cee79a595c376ac9f8c8bb205cf690beca4662e70829462fc4cf32eae6 not found: ID does not exist" Nov 03 16:47:54.096098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:54.096065 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk"] Nov 03 16:47:54.147098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:54.147065 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-2-w4-2bb620-kserve-58b85c9996-jpntk"] Nov 03 16:47:56.005038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:56.005001 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" path="/var/lib/kubelet/pods/40091c43-f73f-4b47-b221-9567bb1670ee/volumes" Nov 03 16:47:56.486790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:47:56.484573 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:00.602134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602101 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj"] Nov 03 16:48:00.602702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602687 5946 cpu_manager.go:414] "containerMap: removing container" podUID="411c49f0-8316-415a-af3e-b97b8a250e12" containerName="revoke-keys" Nov 03 16:48:00.602702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602702 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411c49f0-8316-415a-af3e-b97b8a250e12" containerName="revoke-keys" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602717 5946 cpu_manager.go:414] "containerMap: removing container" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602722 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602740 5946 cpu_manager.go:414] "containerMap: removing container" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602745 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602765 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602769 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602778 5946 cpu_manager.go:414] "containerMap: removing container" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602782 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" Nov 03 16:48:00.602792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602792 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" Nov 03 16:48:00.603061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602809 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" Nov 03 16:48:00.603061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602909 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="411c49f0-8316-415a-af3e-b97b8a250e12" containerName="revoke-keys" Nov 03 16:48:00.603061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602920 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" Nov 03 16:48:00.603061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602927 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bdfccd-eb96-409f-9f80-d4cecc7a6a5f" containerName="istio-proxy" Nov 03 16:48:00.603061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602936 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" Nov 03 16:48:00.603061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602943 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="40091c43-f73f-4b47-b221-9567bb1670ee" containerName="main" Nov 03 16:48:00.603061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.602948 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f08a11-6683-4b7b-a297-7a75aa981220" containerName="istio-proxy" Nov 03 16:48:00.606499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.606484 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.656572 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.656537 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9"] Nov 03 16:48:00.660774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.660752 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.687014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.686973 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.687232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.687192 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzb5x\" (UniqueName: \"kubernetes.io/projected/82b5c8b0-0ffa-4e49-8a43-07f390c29056-kube-api-access-nzb5x\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.687342 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.687325 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.703897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.703865 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:48:00.707252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.707221 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj"] Nov 03 16:48:00.712705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.712685 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:48:00.712705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.712699 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:48:00.712905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.712699 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:48:00.722321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.722293 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9"] Nov 03 16:48:00.788796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.788762 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.788954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.788845 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rc8c\" (UniqueName: \"kubernetes.io/projected/9f057cd7-60d5-4e98-ad80-627e7b06eca4-kube-api-access-6rc8c\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.789014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.788975 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzb5x\" (UniqueName: \"kubernetes.io/projected/82b5c8b0-0ffa-4e49-8a43-07f390c29056-kube-api-access-nzb5x\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.789055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.789038 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.789094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.789075 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.789129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.789106 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.789685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.789655 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.791577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.791554 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.849936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.849905 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzb5x\" (UniqueName: \"kubernetes.io/projected/82b5c8b0-0ffa-4e49-8a43-07f390c29056-kube-api-access-nzb5x\") pod \"maas-api-revoke-sub-6cc577fe-fw6kj\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.890919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.890834 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.891087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.890924 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rc8c\" (UniqueName: \"kubernetes.io/projected/9f057cd7-60d5-4e98-ad80-627e7b06eca4-kube-api-access-6rc8c\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.891156 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.891094 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.891689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.891668 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.893315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.893293 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.920726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.920696 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:00.952020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.951991 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rc8c\" (UniqueName: \"kubernetes.io/projected/9f057cd7-60d5-4e98-ad80-627e7b06eca4-kube-api-access-6rc8c\") pod \"maas-api-revoke-sub-f2ff2e23-f2zb9\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:00.979817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:00.978049 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:01.305257 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:01.305226 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj"] Nov 03 16:48:01.305698 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:48:01.305673 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b5c8b0_0ffa_4e49_8a43_07f390c29056.slice/crio-0292ae063494d0f6296f20dcdd340b318a099ec00c5b69a236367f66d404ea95 WatchSource:0}: Error finding container 0292ae063494d0f6296f20dcdd340b318a099ec00c5b69a236367f66d404ea95: Status 404 returned error can't find the container with id 0292ae063494d0f6296f20dcdd340b318a099ec00c5b69a236367f66d404ea95 Nov 03 16:48:01.408590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:01.408560 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9"] Nov 03 16:48:01.409037 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:48:01.409012 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f057cd7_60d5_4e98_ad80_627e7b06eca4.slice/crio-aad5362cccbe6a4b44aa6378bf174ce942e9bd9e1d9b74ffd3b047e602fd6fce WatchSource:0}: Error finding container aad5362cccbe6a4b44aa6378bf174ce942e9bd9e1d9b74ffd3b047e602fd6fce: Status 404 returned error can't find the container with id aad5362cccbe6a4b44aa6378bf174ce942e9bd9e1d9b74ffd3b047e602fd6fce Nov 03 16:48:01.485528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:01.485492 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:01.986975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:01.986938 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" event={"ID":"9f057cd7-60d5-4e98-ad80-627e7b06eca4","Type":"ContainerStarted","Data":"cf2157ccdd32e7be78986d814a69c67915fff8a86d59332d1e0765dbe4c82bda"} Nov 03 16:48:01.987442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:01.986999 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" event={"ID":"9f057cd7-60d5-4e98-ad80-627e7b06eca4","Type":"ContainerStarted","Data":"aad5362cccbe6a4b44aa6378bf174ce942e9bd9e1d9b74ffd3b047e602fd6fce"} Nov 03 16:48:01.988493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:01.988468 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" event={"ID":"82b5c8b0-0ffa-4e49-8a43-07f390c29056","Type":"ContainerStarted","Data":"1a056ae853d52f4672220d0a4b7015361627c2736607f52b75b88627ead90de8"} Nov 03 16:48:01.988623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:01.988507 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" event={"ID":"82b5c8b0-0ffa-4e49-8a43-07f390c29056","Type":"ContainerStarted","Data":"0292ae063494d0f6296f20dcdd340b318a099ec00c5b69a236367f66d404ea95"} Nov 03 16:48:02.108130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:02.108029 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" podStartSLOduration=2.108014273 podStartE2EDuration="2.108014273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:02.107953162 +0000 UTC m=+2152.229005991" watchObservedRunningTime="2026-11-03 16:48:02.108014273 +0000 UTC m=+2152.229067095" Nov 03 16:48:02.228928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:02.228858 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" podStartSLOduration=2.228838417 podStartE2EDuration="2.228838417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:02.226881578 +0000 UTC m=+2152.347934402" watchObservedRunningTime="2026-11-03 16:48:02.228838417 +0000 UTC m=+2152.349891239" Nov 03 16:48:02.998752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:02.998710 5946 generic.go:356] "Generic (PLEG): container finished" podID="9f057cd7-60d5-4e98-ad80-627e7b06eca4" containerID="cf2157ccdd32e7be78986d814a69c67915fff8a86d59332d1e0765dbe4c82bda" exitCode=0 Nov 03 16:48:02.999275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:02.998781 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" event={"ID":"9f057cd7-60d5-4e98-ad80-627e7b06eca4","Type":"ContainerDied","Data":"cf2157ccdd32e7be78986d814a69c67915fff8a86d59332d1e0765dbe4c82bda"} Nov 03 16:48:03.000493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:03.000465 5946 generic.go:356] "Generic (PLEG): container finished" podID="82b5c8b0-0ffa-4e49-8a43-07f390c29056" containerID="1a056ae853d52f4672220d0a4b7015361627c2736607f52b75b88627ead90de8" exitCode=0 Nov 03 16:48:03.000635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:03.000513 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" event={"ID":"82b5c8b0-0ffa-4e49-8a43-07f390c29056","Type":"ContainerDied","Data":"1a056ae853d52f4672220d0a4b7015361627c2736607f52b75b88627ead90de8"} Nov 03 16:48:04.208697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.208669 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:04.212618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.212599 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:04.338818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.338717 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-openshift-ca\") pod \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " Nov 03 16:48:04.339034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.338850 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b5c8b0-0ffa-4e49-8a43-07f390c29056-kube-api-access-nzb5x\" (UniqueName: \"kubernetes.io/projected/82b5c8b0-0ffa-4e49-8a43-07f390c29056-kube-api-access-nzb5x\") pod \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " Nov 03 16:48:04.339034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.339006 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f057cd7-60d5-4e98-ad80-627e7b06eca4-kube-api-access-6rc8c\" (UniqueName: \"kubernetes.io/projected/9f057cd7-60d5-4e98-ad80-627e7b06eca4-kube-api-access-6rc8c\") pod \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " Nov 03 16:48:04.339225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.339072 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-xks-ca\") pod \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " Nov 03 16:48:04.339225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.339134 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-openshift-ca\") pod \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\" (UID: \"9f057cd7-60d5-4e98-ad80-627e7b06eca4\") " Nov 03 16:48:04.339395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.339245 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-xks-ca\") pod \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\" (UID: \"82b5c8b0-0ffa-4e49-8a43-07f390c29056\") " Nov 03 16:48:04.339832 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.339804 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-openshift-ca" pod "82b5c8b0-0ffa-4e49-8a43-07f390c29056" (UID: "82b5c8b0-0ffa-4e49-8a43-07f390c29056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:04.340181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.340120 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-openshift-ca" pod "9f057cd7-60d5-4e98-ad80-627e7b06eca4" (UID: "9f057cd7-60d5-4e98-ad80-627e7b06eca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:04.342886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.342858 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-xks-ca" pod "9f057cd7-60d5-4e98-ad80-627e7b06eca4" (UID: "9f057cd7-60d5-4e98-ad80-627e7b06eca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:04.342996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.342895 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f057cd7-60d5-4e98-ad80-627e7b06eca4-kube-api-access-6rc8c" pod "9f057cd7-60d5-4e98-ad80-627e7b06eca4" (UID: "9f057cd7-60d5-4e98-ad80-627e7b06eca4"). InnerVolumeSpecName "kube-api-access-6rc8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:04.342996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.342931 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b5c8b0-0ffa-4e49-8a43-07f390c29056-kube-api-access-nzb5x" pod "82b5c8b0-0ffa-4e49-8a43-07f390c29056" (UID: "82b5c8b0-0ffa-4e49-8a43-07f390c29056"). InnerVolumeSpecName "kube-api-access-nzb5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:04.343383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.343355 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-xks-ca" pod "82b5c8b0-0ffa-4e49-8a43-07f390c29056" (UID: "82b5c8b0-0ffa-4e49-8a43-07f390c29056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:04.440911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.440871 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rc8c\" (UniqueName: \"kubernetes.io/projected/9f057cd7-60d5-4e98-ad80-627e7b06eca4-kube-api-access-6rc8c\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:04.440911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.440905 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:04.440911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.440915 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f057cd7-60d5-4e98-ad80-627e7b06eca4-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:04.441139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.440924 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:04.441139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.440933 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b5c8b0-0ffa-4e49-8a43-07f390c29056-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:04.441139 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:04.440942 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzb5x\" (UniqueName: \"kubernetes.io/projected/82b5c8b0-0ffa-4e49-8a43-07f390c29056-kube-api-access-nzb5x\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:05.016499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:05.016462 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" Nov 03 16:48:05.016693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:05.016502 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9" event={"ID":"9f057cd7-60d5-4e98-ad80-627e7b06eca4","Type":"ContainerDied","Data":"aad5362cccbe6a4b44aa6378bf174ce942e9bd9e1d9b74ffd3b047e602fd6fce"} Nov 03 16:48:05.016693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:05.016550 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad5362cccbe6a4b44aa6378bf174ce942e9bd9e1d9b74ffd3b047e602fd6fce" Nov 03 16:48:05.017926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:05.017906 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" Nov 03 16:48:05.018054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:05.017939 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj" event={"ID":"82b5c8b0-0ffa-4e49-8a43-07f390c29056","Type":"ContainerDied","Data":"0292ae063494d0f6296f20dcdd340b318a099ec00c5b69a236367f66d404ea95"} Nov 03 16:48:05.018054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:05.017981 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0292ae063494d0f6296f20dcdd340b318a099ec00c5b69a236367f66d404ea95" Nov 03 16:48:06.191397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:06.191362 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:09.340600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:09.340564 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:48:10.969305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:10.969267 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:12.718713 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:48:12.718670 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:48:13.477027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:13.476970 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:48:13.477689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:13.477651 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 16:48:13.478480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:13.477893 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:48:13.563233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:13.561669 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:48:13.896394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:13.896343 5946 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:48:15.633293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:15.633261 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:17.414513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:17.414428 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:17.619688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:17.619647 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:17.813242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:17.813188 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:17.938502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:17.938457 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt"] Nov 03 16:48:17.943867 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:17.943842 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23c440-kwbpt"] Nov 03 16:48:18.006222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:18.006102 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:18.027639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:18.027601 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb63bf45-9ace-41ed-a1d9-48b67b440ba8" path="/var/lib/kubelet/pods/fb63bf45-9ace-41ed-a1d9-48b67b440ba8/volumes" Nov 03 16:48:18.351018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:18.350974 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:19.866396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:19.866363 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:21.230857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:21.230823 5946 generic.go:356] "Generic (PLEG): container finished" podID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerID="23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a" exitCode=0 Nov 03 16:48:21.231442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:21.230877 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" event={"ID":"96423ef9-ea37-4e35-ad23-015b1bf7598a","Type":"ContainerDied","Data":"23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a"} Nov 03 16:48:21.231522 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:21.231449 5946 scope.go:122] "RemoveContainer" containerID="23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a" Nov 03 16:48:21.233333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:21.233305 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:48:21.233409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:21.233375 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:48:21.233446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:21.233408 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:48:21.876349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:21.876315 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:22.271941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:22.271907 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" event={"ID":"96423ef9-ea37-4e35-ad23-015b1bf7598a","Type":"ContainerStarted","Data":"02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f"} Nov 03 16:48:22.800296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:22.795948 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:48:22.800296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:22.798564 5946 patch_prober.go:29] interesting pod/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 03 16:48:22.800296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:22.798636 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 03 16:48:23.465306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.465269 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:23.705895 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.705828 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:23.795870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.795836 5946 patch_prober.go:29] interesting pod/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 03 16:48:23.796058 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.795900 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 03 16:48:23.903582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.903550 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:23.937233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.934897 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f"] Nov 03 16:48:23.940062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.940040 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc4dcd-dt58f"] Nov 03 16:48:23.943446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:23.943427 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:48:24.005719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:24.005694 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f4e9f8-6817-4913-b142-4b68439cd2d5" path="/var/lib/kubelet/pods/b6f4e9f8-6817-4913-b142-4b68439cd2d5/volumes" Nov 03 16:48:24.796407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:24.796370 5946 patch_prober.go:29] interesting pod/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 03 16:48:24.796875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:24.796441 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 03 16:48:25.795990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:25.795954 5946 patch_prober.go:29] interesting pod/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 03 16:48:25.796175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:25.796016 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 03 16:48:26.285589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:26.285522 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:26.486271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:26.486198 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:26.694180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:26.692878 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:26.796659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:26.796605 5946 patch_prober.go:29] interesting pod/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 03 16:48:26.796870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:26.796682 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 03 16:48:27.405284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.405242 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r"] Nov 03 16:48:27.405871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.405855 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9f057cd7-60d5-4e98-ad80-627e7b06eca4" containerName="revoke-keys" Nov 03 16:48:27.405916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.405873 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f057cd7-60d5-4e98-ad80-627e7b06eca4" containerName="revoke-keys" Nov 03 16:48:27.405916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.405884 5946 cpu_manager.go:414] "containerMap: removing container" podUID="82b5c8b0-0ffa-4e49-8a43-07f390c29056" containerName="revoke-keys" Nov 03 16:48:27.405916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.405889 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b5c8b0-0ffa-4e49-8a43-07f390c29056" containerName="revoke-keys" Nov 03 16:48:27.406036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.406025 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b5c8b0-0ffa-4e49-8a43-07f390c29056" containerName="revoke-keys" Nov 03 16:48:27.406073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.406052 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f057cd7-60d5-4e98-ad80-627e7b06eca4" containerName="revoke-keys" Nov 03 16:48:27.408617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.408601 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.471617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.471579 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r"] Nov 03 16:48:27.485173 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.485130 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.485418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.485242 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.485510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.485428 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkrh\" (UniqueName: \"kubernetes.io/projected/f6ae9b93-97a5-498c-8189-dff362071fce-kube-api-access-lbkrh\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.516427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.516397 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:48:27.516427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.516407 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:48:27.516427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.516409 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:48:27.516680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.516446 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:48:27.586379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.586347 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkrh\" (UniqueName: \"kubernetes.io/projected/f6ae9b93-97a5-498c-8189-dff362071fce-kube-api-access-lbkrh\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.586580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.586546 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.586655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.586579 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.587381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.587352 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.589477 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.589454 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.648448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.648416 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkrh\" (UniqueName: \"kubernetes.io/projected/f6ae9b93-97a5-498c-8189-dff362071fce-kube-api-access-lbkrh\") pod \"maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.723800 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.723721 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:27.808503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.808464 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:48:27.810416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.810392 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:48:27.928757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:27.928726 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:28.108141 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:28.108095 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r"] Nov 03 16:48:28.108372 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:48:28.108344 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ae9b93_97a5_498c_8189_dff362071fce.slice/crio-bbd736d79f68da37e9b0e962ccee42eae1bc3cd9396e573baab99dc0ae6ce052 WatchSource:0}: Error finding container bbd736d79f68da37e9b0e962ccee42eae1bc3cd9396e573baab99dc0ae6ce052: Status 404 returned error can't find the container with id bbd736d79f68da37e9b0e962ccee42eae1bc3cd9396e573baab99dc0ae6ce052 Nov 03 16:48:28.317493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:28.317445 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" event={"ID":"f6ae9b93-97a5-498c-8189-dff362071fce","Type":"ContainerStarted","Data":"bbd736d79f68da37e9b0e962ccee42eae1bc3cd9396e573baab99dc0ae6ce052"} Nov 03 16:48:28.775081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:28.775047 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:29.326868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:29.326782 5946 generic.go:356] "Generic (PLEG): container finished" podID="f6ae9b93-97a5-498c-8189-dff362071fce" containerID="7201eafb8266f6feb998aa627afc4691424e3b9eb6e42b69fb27e45a2e0b4615" exitCode=0 Nov 03 16:48:29.327017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:29.326863 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" event={"ID":"f6ae9b93-97a5-498c-8189-dff362071fce","Type":"ContainerDied","Data":"7201eafb8266f6feb998aa627afc4691424e3b9eb6e42b69fb27e45a2e0b4615"} Nov 03 16:48:30.508060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.508034 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:30.628414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.628295 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-openshift-ca\") pod \"f6ae9b93-97a5-498c-8189-dff362071fce\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " Nov 03 16:48:30.628610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.628567 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-xks-ca\") pod \"f6ae9b93-97a5-498c-8189-dff362071fce\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " Nov 03 16:48:30.628674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.628634 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ae9b93-97a5-498c-8189-dff362071fce-kube-api-access-lbkrh\" (UniqueName: \"kubernetes.io/projected/f6ae9b93-97a5-498c-8189-dff362071fce-kube-api-access-lbkrh\") pod \"f6ae9b93-97a5-498c-8189-dff362071fce\" (UID: \"f6ae9b93-97a5-498c-8189-dff362071fce\") " Nov 03 16:48:30.628723 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.628690 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-openshift-ca" pod "f6ae9b93-97a5-498c-8189-dff362071fce" (UID: "f6ae9b93-97a5-498c-8189-dff362071fce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:30.630827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.629176 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:30.631945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.631730 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-xks-ca" pod "f6ae9b93-97a5-498c-8189-dff362071fce" (UID: "f6ae9b93-97a5-498c-8189-dff362071fce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:30.631945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.631752 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ae9b93-97a5-498c-8189-dff362071fce-kube-api-access-lbkrh" pod "f6ae9b93-97a5-498c-8189-dff362071fce" (UID: "f6ae9b93-97a5-498c-8189-dff362071fce"). InnerVolumeSpecName "kube-api-access-lbkrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:30.730517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.730485 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbkrh\" (UniqueName: \"kubernetes.io/projected/f6ae9b93-97a5-498c-8189-dff362071fce-kube-api-access-lbkrh\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:30.730517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:30.730515 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ae9b93-97a5-498c-8189-dff362071fce-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:31.342597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:31.342568 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" Nov 03 16:48:31.342778 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:31.342604 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r" event={"ID":"f6ae9b93-97a5-498c-8189-dff362071fce","Type":"ContainerDied","Data":"bbd736d79f68da37e9b0e962ccee42eae1bc3cd9396e573baab99dc0ae6ce052"} Nov 03 16:48:31.342778 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:31.342677 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbd736d79f68da37e9b0e962ccee42eae1bc3cd9396e573baab99dc0ae6ce052" Nov 03 16:48:33.671041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:33.671002 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:34.352165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:34.352132 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:36.803237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:36.803169 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:38.885166 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:38.885132 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75"] Nov 03 16:48:39.246950 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:39.246918 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:39.447771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:39.447730 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:39.639072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:39.638978 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:39.742955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:39.742921 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82"] Nov 03 16:48:39.749368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:39.749339 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f58cf-hqm82"] Nov 03 16:48:40.005725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:40.005692 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82783799-5fb1-44dc-9b0c-bb36a4de9c46" path="/var/lib/kubelet/pods/82783799-5fb1-44dc-9b0c-bb36a4de9c46/volumes" Nov 03 16:48:40.125812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:40.125779 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:42.586139 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:48:42.586103 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:48:42.975323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:42.975146 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:43.177444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:43.177415 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:43.401136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:43.401101 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:43.885830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:43.885785 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" podUID="0e14ac8d-1359-4333-9cd5-098963a92698" containerName="payload-processing" containerID="cri-o://1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963" gracePeriod=25 Nov 03 16:48:44.113852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.113819 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh"] Nov 03 16:48:44.114817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.114793 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f6ae9b93-97a5-498c-8189-dff362071fce" containerName="revoke-keys" Nov 03 16:48:44.114817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.114815 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ae9b93-97a5-498c-8189-dff362071fce" containerName="revoke-keys" Nov 03 16:48:44.115018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.114977 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ae9b93-97a5-498c-8189-dff362071fce" containerName="revoke-keys" Nov 03 16:48:44.118088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.118066 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.161304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.161278 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:48:44.178558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.178524 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh"] Nov 03 16:48:44.216286 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.216249 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-kube-api-access-rh2f2\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.216459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.216426 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.216546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.216532 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.224855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.224826 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:48:44.225035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.224870 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:48:44.225035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.224953 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:48:44.225035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.224998 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:48:44.317523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.317432 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e14ac8d-1359-4333-9cd5-098963a92698-kube-api-access-nmdz5\" (UniqueName: \"kubernetes.io/projected/0e14ac8d-1359-4333-9cd5-098963a92698-kube-api-access-nmdz5\") pod \"0e14ac8d-1359-4333-9cd5-098963a92698\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " Nov 03 16:48:44.317692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.317607 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e14ac8d-1359-4333-9cd5-098963a92698-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e14ac8d-1359-4333-9cd5-098963a92698-plugins-config-volume\") pod \"0e14ac8d-1359-4333-9cd5-098963a92698\" (UID: \"0e14ac8d-1359-4333-9cd5-098963a92698\") " Nov 03 16:48:44.317834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.317818 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.317927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.317890 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.317991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.317982 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-kube-api-access-rh2f2\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.318049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.318023 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e14ac8d-1359-4333-9cd5-098963a92698-plugins-config-volume" pod "0e14ac8d-1359-4333-9cd5-098963a92698" (UID: "0e14ac8d-1359-4333-9cd5-098963a92698"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:44.318248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.318196 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e14ac8d-1359-4333-9cd5-098963a92698-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:44.318970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.318948 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.320027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.319997 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e14ac8d-1359-4333-9cd5-098963a92698-kube-api-access-nmdz5" pod "0e14ac8d-1359-4333-9cd5-098963a92698" (UID: "0e14ac8d-1359-4333-9cd5-098963a92698"). InnerVolumeSpecName "kube-api-access-nmdz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:44.320361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.320342 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.379244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.379176 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-kube-api-access-rh2f2\") pod \"maas-api-revoke-keys-e2e-api-b-0fda73-bwclh\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.419704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.419667 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdz5\" (UniqueName: \"kubernetes.io/projected/0e14ac8d-1359-4333-9cd5-098963a92698-kube-api-access-nmdz5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:44.434324 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.434292 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:44.513739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.511474 5946 generic.go:356] "Generic (PLEG): container finished" podID="0e14ac8d-1359-4333-9cd5-098963a92698" containerID="1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963" exitCode=0 Nov 03 16:48:44.513739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.511598 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" Nov 03 16:48:44.513739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.511740 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" event={"ID":"0e14ac8d-1359-4333-9cd5-098963a92698","Type":"ContainerDied","Data":"1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963"} Nov 03 16:48:44.513739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.511779 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75" event={"ID":"0e14ac8d-1359-4333-9cd5-098963a92698","Type":"ContainerDied","Data":"006ade6683b3ae54f76181750aa4285d7331ffa98cc958115bdf3aa3d9d37109"} Nov 03 16:48:44.513739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.511806 5946 scope.go:122] "RemoveContainer" containerID="1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963" Nov 03 16:48:44.535094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.535066 5946 scope.go:122] "RemoveContainer" containerID="1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963" Nov 03 16:48:44.535580 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:48:44.535553 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963\": container with ID starting with 1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963 not found: ID does not exist" containerID="1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963" Nov 03 16:48:44.535684 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.535591 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963"} err="failed to get container status \"1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963\": rpc error: code = NotFound desc = could not find container \"1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963\": container with ID starting with 1a1a414e4d447edbb1d1de29f678a938f12bd16f258dc516eff10f7bd2a60963 not found: ID does not exist" Nov 03 16:48:44.611447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.611337 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:44.688113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.688079 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75"] Nov 03 16:48:44.705889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.705849 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ea8a5-78f5f6468c-4jv75"] Nov 03 16:48:44.831753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:44.831724 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh"] Nov 03 16:48:44.832560 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:48:44.832531 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cae2190_3e4c_4cb5_8c31_46c89eae2d3f.slice/crio-928c9dbbc127e61e98442160a76d3b35604f834083cb1f606ecabf74b3a1335b WatchSource:0}: Error finding container 928c9dbbc127e61e98442160a76d3b35604f834083cb1f606ecabf74b3a1335b: Status 404 returned error can't find the container with id 928c9dbbc127e61e98442160a76d3b35604f834083cb1f606ecabf74b3a1335b Nov 03 16:48:45.523249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:45.523182 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" event={"ID":"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f","Type":"ContainerStarted","Data":"771a27d7495a5722ede14d83125f5ab6db695947c48abdcfeb4fe04bdb32a353"} Nov 03 16:48:45.523760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:45.523279 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" event={"ID":"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f","Type":"ContainerStarted","Data":"928c9dbbc127e61e98442160a76d3b35604f834083cb1f606ecabf74b3a1335b"} Nov 03 16:48:45.655019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:45.654962 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" podStartSLOduration=1.654946893 podStartE2EDuration="1.654946893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:45.652598244 +0000 UTC m=+2195.773651067" watchObservedRunningTime="2026-11-03 16:48:45.654946893 +0000 UTC m=+2195.775999716" Nov 03 16:48:46.005502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:46.005466 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e14ac8d-1359-4333-9cd5-098963a92698" path="/var/lib/kubelet/pods/0e14ac8d-1359-4333-9cd5-098963a92698/volumes" Nov 03 16:48:46.357817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:46.357733 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:46.566820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:46.565979 5946 generic.go:356] "Generic (PLEG): container finished" podID="6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" containerID="771a27d7495a5722ede14d83125f5ab6db695947c48abdcfeb4fe04bdb32a353" exitCode=0 Nov 03 16:48:46.566820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:46.566109 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" event={"ID":"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f","Type":"ContainerDied","Data":"771a27d7495a5722ede14d83125f5ab6db695947c48abdcfeb4fe04bdb32a353"} Nov 03 16:48:46.583407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:46.576914 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:47.741882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.741852 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:47.872371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.872294 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-kube-api-access-rh2f2\") pod \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " Nov 03 16:48:47.872371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.872341 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-openshift-ca\") pod \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " Nov 03 16:48:47.872573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.872513 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-xks-ca\") pod \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\" (UID: \"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f\") " Nov 03 16:48:47.872775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.872746 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-openshift-ca" pod "6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" (UID: "6cae2190-3e4c-4cb5-8c31-46c89eae2d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:47.873388 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.873365 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:47.875054 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.875025 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-kube-api-access-rh2f2" pod "6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" (UID: "6cae2190-3e4c-4cb5-8c31-46c89eae2d3f"). InnerVolumeSpecName "kube-api-access-rh2f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:47.875155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.875067 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-xks-ca" pod "6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" (UID: "6cae2190-3e4c-4cb5-8c31-46c89eae2d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:47.975104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.975073 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:47.975104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:47.975100 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f-kube-api-access-rh2f2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:48:48.581806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:48.581770 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" Nov 03 16:48:48.581806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:48.581787 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh" event={"ID":"6cae2190-3e4c-4cb5-8c31-46c89eae2d3f","Type":"ContainerDied","Data":"928c9dbbc127e61e98442160a76d3b35604f834083cb1f606ecabf74b3a1335b"} Nov 03 16:48:48.582034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:48.581838 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="928c9dbbc127e61e98442160a76d3b35604f834083cb1f606ecabf74b3a1335b" Nov 03 16:48:48.626501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:48.626472 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:48.818868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:48.818837 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:49.008110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:49.008075 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:49.998445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:49.996116 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:51.115696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:51.115663 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:51.849280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:51.845027 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:52.176475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:52.176368 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:54.941290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:54.941184 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:55.708936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:55.708902 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9"] Nov 03 16:48:55.824675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:55.824635 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm"] Nov 03 16:48:56.134897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:56.134859 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:56.331871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:56.331834 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:56.525611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:56.525574 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:59.870563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:48:59.870481 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:00.074074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:00.074044 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:00.281224 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:00.281178 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:00.709783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:00.709688 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" podUID="01e504b7-fce1-4fc7-95b1-fc968073f8ad" containerName="payload-processing" containerID="cri-o://93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5" gracePeriod=25 Nov 03 16:49:00.825253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:00.825216 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" podUID="c2951083-7d6d-4466-8f76-8f3fcdd047fb" containerName="payload-pre-processing" containerID="cri-o://71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647" gracePeriod=25 Nov 03 16:49:00.978110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:00.978037 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:49:01.099469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.099443 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:49:01.120364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.120329 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e504b7-fce1-4fc7-95b1-fc968073f8ad-kube-api-access-7nvhn\" (UniqueName: \"kubernetes.io/projected/01e504b7-fce1-4fc7-95b1-fc968073f8ad-kube-api-access-7nvhn\") pod \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " Nov 03 16:49:01.120703 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.120678 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e504b7-fce1-4fc7-95b1-fc968073f8ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e504b7-fce1-4fc7-95b1-fc968073f8ad-plugins-config-volume\") pod \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\" (UID: \"01e504b7-fce1-4fc7-95b1-fc968073f8ad\") " Nov 03 16:49:01.122106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.122073 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e504b7-fce1-4fc7-95b1-fc968073f8ad-plugins-config-volume" pod "01e504b7-fce1-4fc7-95b1-fc968073f8ad" (UID: "01e504b7-fce1-4fc7-95b1-fc968073f8ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:01.122733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.122709 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e504b7-fce1-4fc7-95b1-fc968073f8ad-kube-api-access-7nvhn" pod "01e504b7-fce1-4fc7-95b1-fc968073f8ad" (UID: "01e504b7-fce1-4fc7-95b1-fc968073f8ad"). InnerVolumeSpecName "kube-api-access-7nvhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:01.222292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.222250 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2951083-7d6d-4466-8f76-8f3fcdd047fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2951083-7d6d-4466-8f76-8f3fcdd047fb-plugins-config-volume\") pod \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " Nov 03 16:49:01.222484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.222381 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2951083-7d6d-4466-8f76-8f3fcdd047fb-kube-api-access-k5nww\" (UniqueName: \"kubernetes.io/projected/c2951083-7d6d-4466-8f76-8f3fcdd047fb-kube-api-access-k5nww\") pod \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\" (UID: \"c2951083-7d6d-4466-8f76-8f3fcdd047fb\") " Nov 03 16:49:01.222708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.222683 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2951083-7d6d-4466-8f76-8f3fcdd047fb-plugins-config-volume" pod "c2951083-7d6d-4466-8f76-8f3fcdd047fb" (UID: "c2951083-7d6d-4466-8f76-8f3fcdd047fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:01.222979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.222951 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e504b7-fce1-4fc7-95b1-fc968073f8ad-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:01.222979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.222981 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nvhn\" (UniqueName: \"kubernetes.io/projected/01e504b7-fce1-4fc7-95b1-fc968073f8ad-kube-api-access-7nvhn\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:01.223170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.222999 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2951083-7d6d-4466-8f76-8f3fcdd047fb-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:01.225733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.225706 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2951083-7d6d-4466-8f76-8f3fcdd047fb-kube-api-access-k5nww" pod "c2951083-7d6d-4466-8f76-8f3fcdd047fb" (UID: "c2951083-7d6d-4466-8f76-8f3fcdd047fb"). InnerVolumeSpecName "kube-api-access-k5nww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:01.324382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.324347 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5nww\" (UniqueName: \"kubernetes.io/projected/c2951083-7d6d-4466-8f76-8f3fcdd047fb-kube-api-access-k5nww\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:01.736717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.736686 5946 generic.go:356] "Generic (PLEG): container finished" podID="c2951083-7d6d-4466-8f76-8f3fcdd047fb" containerID="71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647" exitCode=0 Nov 03 16:49:01.736843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.736790 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" Nov 03 16:49:01.736843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.736796 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" event={"ID":"c2951083-7d6d-4466-8f76-8f3fcdd047fb","Type":"ContainerDied","Data":"71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647"} Nov 03 16:49:01.736918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.736851 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm" event={"ID":"c2951083-7d6d-4466-8f76-8f3fcdd047fb","Type":"ContainerDied","Data":"47601d6b4fbb121a5f997ef557868ed6d47e9d930a71e5192ad979a2a2882e23"} Nov 03 16:49:01.736918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.736880 5946 scope.go:122] "RemoveContainer" containerID="71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647" Nov 03 16:49:01.738263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.738149 5946 generic.go:356] "Generic (PLEG): container finished" podID="01e504b7-fce1-4fc7-95b1-fc968073f8ad" containerID="93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5" exitCode=0 Nov 03 16:49:01.738263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.738237 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" Nov 03 16:49:01.738412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.738263 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" event={"ID":"01e504b7-fce1-4fc7-95b1-fc968073f8ad","Type":"ContainerDied","Data":"93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5"} Nov 03 16:49:01.738412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.738305 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9" event={"ID":"01e504b7-fce1-4fc7-95b1-fc968073f8ad","Type":"ContainerDied","Data":"6e3ffcb1bd86646fac05f7a408a50916034bcd1c3f6151ad7a21fc1e0a2ab5d8"} Nov 03 16:49:01.749989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.749970 5946 scope.go:122] "RemoveContainer" containerID="71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647" Nov 03 16:49:01.750396 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:49:01.750356 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647\": container with ID starting with 71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647 not found: ID does not exist" containerID="71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647" Nov 03 16:49:01.750608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.750570 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647"} err="failed to get container status \"71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647\": rpc error: code = NotFound desc = could not find container \"71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647\": container with ID starting with 71c75ace182db1827c43062bc3c2fdeaa8cac305ec8beb61733dbdb540dc5647 not found: ID does not exist" Nov 03 16:49:01.750717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.750612 5946 scope.go:122] "RemoveContainer" containerID="93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5" Nov 03 16:49:01.764313 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.764283 5946 scope.go:122] "RemoveContainer" containerID="93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5" Nov 03 16:49:01.764698 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:49:01.764669 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5\": container with ID starting with 93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5 not found: ID does not exist" containerID="93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5" Nov 03 16:49:01.764879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.764711 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5"} err="failed to get container status \"93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5\": rpc error: code = NotFound desc = could not find container \"93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5\": container with ID starting with 93aa928d63cdf13d23ab18ff111e4dca090e49207bec84a577fe190f54d36af5 not found: ID does not exist" Nov 03 16:49:01.912015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.911986 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9"] Nov 03 16:49:01.962368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:01.962336 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fda73-54bc5b9fbc-jtpn9"] Nov 03 16:49:02.004862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:02.004830 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e504b7-fce1-4fc7-95b1-fc968073f8ad" path="/var/lib/kubelet/pods/01e504b7-fce1-4fc7-95b1-fc968073f8ad/volumes" Nov 03 16:49:02.041685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:02.041649 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm"] Nov 03 16:49:02.045215 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:02.045180 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fda73-9fd748899-r8prm"] Nov 03 16:49:04.004261 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:04.004230 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2951083-7d6d-4466-8f76-8f3fcdd047fb" path="/var/lib/kubelet/pods/c2951083-7d6d-4466-8f76-8f3fcdd047fb/volumes" Nov 03 16:49:04.974343 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:04.974289 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:05.167455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:05.167410 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:05.364666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:05.364629 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:09.948065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:09.948027 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:11.532451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:11.532417 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:11.848857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:11.848767 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:12.074114 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:12.074085 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44"] Nov 03 16:49:12.082647 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:12.082619 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:12.125421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:12.125332 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb494a-wtx44"] Nov 03 16:49:12.509482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:12.509451 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:12.586112 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:49:12.586079 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:49:14.004774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:14.004741 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345e17c0-583a-43dc-93ec-5c62ee66f8f9" path="/var/lib/kubelet/pods/345e17c0-583a-43dc-93ec-5c62ee66f8f9/volumes" Nov 03 16:49:16.984701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:16.984655 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:19.376484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.376442 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk"] Nov 03 16:49:19.377105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377086 5946 cpu_manager.go:414] "containerMap: removing container" podUID="0e14ac8d-1359-4333-9cd5-098963a92698" containerName="payload-processing" Nov 03 16:49:19.377105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377106 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e14ac8d-1359-4333-9cd5-098963a92698" containerName="payload-processing" Nov 03 16:49:19.377299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377132 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c2951083-7d6d-4466-8f76-8f3fcdd047fb" containerName="payload-pre-processing" Nov 03 16:49:19.377299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377138 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2951083-7d6d-4466-8f76-8f3fcdd047fb" containerName="payload-pre-processing" Nov 03 16:49:19.377299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377161 5946 cpu_manager.go:414] "containerMap: removing container" podUID="6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" containerName="revoke-keys" Nov 03 16:49:19.377299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377166 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" containerName="revoke-keys" Nov 03 16:49:19.377299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377172 5946 cpu_manager.go:414] "containerMap: removing container" podUID="01e504b7-fce1-4fc7-95b1-fc968073f8ad" containerName="payload-processing" Nov 03 16:49:19.377299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377177 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e504b7-fce1-4fc7-95b1-fc968073f8ad" containerName="payload-processing" Nov 03 16:49:19.377299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377293 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2951083-7d6d-4466-8f76-8f3fcdd047fb" containerName="payload-pre-processing" Nov 03 16:49:19.377567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377308 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" containerName="revoke-keys" Nov 03 16:49:19.377567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377317 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e504b7-fce1-4fc7-95b1-fc968073f8ad" containerName="payload-processing" Nov 03 16:49:19.377567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.377327 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e14ac8d-1359-4333-9cd5-098963a92698" containerName="payload-processing" Nov 03 16:49:19.382475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.382457 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.441777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.441745 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk"] Nov 03 16:49:19.479739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.479370 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxkv\" (UniqueName: \"kubernetes.io/projected/7cede107-03a3-40fe-b87f-2b91928d568f-kube-api-access-dqxkv\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.479739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.479431 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.479739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.479588 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.488351 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.488319 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:49:19.488528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.488475 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:49:19.488528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.488338 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:49:19.488528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.488517 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:49:19.581014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.580984 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.581198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.581162 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxkv\" (UniqueName: \"kubernetes.io/projected/7cede107-03a3-40fe-b87f-2b91928d568f-kube-api-access-dqxkv\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.581198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.581191 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.581891 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.581865 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.583499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.583479 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.642132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.642049 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxkv\" (UniqueName: \"kubernetes.io/projected/7cede107-03a3-40fe-b87f-2b91928d568f-kube-api-access-dqxkv\") pod \"maas-api-revoke-sub-79a8d6b0-kd5wk\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.696040 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.696010 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:19.963073 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:19.962997 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:20.010674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.010612 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c"] Nov 03 16:49:20.018593 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.018564 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.072571 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.072530 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c"] Nov 03 16:49:20.123336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.123308 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk"] Nov 03 16:49:20.124527 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:20.124488 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cede107_03a3_40fe_b87f_2b91928d568f.slice/crio-e19b91dbf452a010d3db2d8091ec953dda54dac192a03e1939bb150514eb5866 WatchSource:0}: Error finding container e19b91dbf452a010d3db2d8091ec953dda54dac192a03e1939bb150514eb5866: Status 404 returned error can't find the container with id e19b91dbf452a010d3db2d8091ec953dda54dac192a03e1939bb150514eb5866 Nov 03 16:49:20.156667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.156635 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:20.192252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.190709 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.192252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.190856 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrkg\" (UniqueName: \"kubernetes.io/projected/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-kube-api-access-czrkg\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.192252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.191038 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.293194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.293155 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.293463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.293410 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.293463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.293440 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrkg\" (UniqueName: \"kubernetes.io/projected/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-kube-api-access-czrkg\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.294048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.294028 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.314489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.314456 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.364234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.360694 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:20.364234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.363245 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrkg\" (UniqueName: \"kubernetes.io/projected/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-kube-api-access-czrkg\") pod \"maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.633082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.633041 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:20.904916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.904831 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" event={"ID":"7cede107-03a3-40fe-b87f-2b91928d568f","Type":"ContainerStarted","Data":"a067650cde49b858a391c59da9a6d8e81202607af9e59f08f2384a7d2917b796"} Nov 03 16:49:20.905090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:20.904931 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" event={"ID":"7cede107-03a3-40fe-b87f-2b91928d568f","Type":"ContainerStarted","Data":"e19b91dbf452a010d3db2d8091ec953dda54dac192a03e1939bb150514eb5866"} Nov 03 16:49:21.027066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.027033 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c"] Nov 03 16:49:21.028179 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:21.028149 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04de1a84_b1e2_4fe1_b1fa_186f4a60e4b6.slice/crio-984a5a17f9b3c9fc561e3ea25970c10d279f63ac4a913e7255d6e9d0a7bbcff0 WatchSource:0}: Error finding container 984a5a17f9b3c9fc561e3ea25970c10d279f63ac4a913e7255d6e9d0a7bbcff0: Status 404 returned error can't find the container with id 984a5a17f9b3c9fc561e3ea25970c10d279f63ac4a913e7255d6e9d0a7bbcff0 Nov 03 16:49:21.040554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.040528 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl"] Nov 03 16:49:21.043155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.043114 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" podStartSLOduration=2.043101423 podStartE2EDuration="2.043101423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:21.035078593 +0000 UTC m=+2231.156131416" watchObservedRunningTime="2026-11-03 16:49:21.043101423 +0000 UTC m=+2231.164154245" Nov 03 16:49:21.045890 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.045872 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.106542 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.106515 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093780f-d756-4959-b202-b772c21aed17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.106687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.106669 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cgzn\" (UniqueName: \"kubernetes.io/projected/e093780f-d756-4959-b202-b772c21aed17-kube-api-access-5cgzn\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.106727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.106717 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093780f-d756-4959-b202-b772c21aed17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.146246 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.146180 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl"] Nov 03 16:49:21.207726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.207633 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgzn\" (UniqueName: \"kubernetes.io/projected/e093780f-d756-4959-b202-b772c21aed17-kube-api-access-5cgzn\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.207726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.207704 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093780f-d756-4959-b202-b772c21aed17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.207962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.207816 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093780f-d756-4959-b202-b772c21aed17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.208537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.208515 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093780f-d756-4959-b202-b772c21aed17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.214621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.214594 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093780f-d756-4959-b202-b772c21aed17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.267720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.267686 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cgzn\" (UniqueName: \"kubernetes.io/projected/e093780f-d756-4959-b202-b772c21aed17-kube-api-access-5cgzn\") pod \"maas-api-revoke-keys-e2e-api-b-66f145-ss6bl\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.459245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.459161 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:21.703159 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.703120 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:21.847702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.847675 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl"] Nov 03 16:49:21.849403 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:21.849373 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode093780f_d756_4959_b202_b772c21aed17.slice/crio-24fc06637ce51c90025f2c522576a92a9921f07c099970e3870ee71b59c15a0a WatchSource:0}: Error finding container 24fc06637ce51c90025f2c522576a92a9921f07c099970e3870ee71b59c15a0a: Status 404 returned error can't find the container with id 24fc06637ce51c90025f2c522576a92a9921f07c099970e3870ee71b59c15a0a Nov 03 16:49:21.924045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.922708 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:21.928707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.928668 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" event={"ID":"e093780f-d756-4959-b202-b772c21aed17","Type":"ContainerStarted","Data":"24fc06637ce51c90025f2c522576a92a9921f07c099970e3870ee71b59c15a0a"} Nov 03 16:49:21.932439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.932405 5946 generic.go:356] "Generic (PLEG): container finished" podID="7cede107-03a3-40fe-b87f-2b91928d568f" containerID="a067650cde49b858a391c59da9a6d8e81202607af9e59f08f2384a7d2917b796" exitCode=0 Nov 03 16:49:21.932629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.932478 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" event={"ID":"7cede107-03a3-40fe-b87f-2b91928d568f","Type":"ContainerDied","Data":"a067650cde49b858a391c59da9a6d8e81202607af9e59f08f2384a7d2917b796"} Nov 03 16:49:21.938049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.935127 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" event={"ID":"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6","Type":"ContainerStarted","Data":"edde3b0db4c09d8eb38f03ccb2eaa2b2271ccd4acbc95ea89728e61f17f4323f"} Nov 03 16:49:21.938049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:21.935187 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" event={"ID":"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6","Type":"ContainerStarted","Data":"984a5a17f9b3c9fc561e3ea25970c10d279f63ac4a913e7255d6e9d0a7bbcff0"} Nov 03 16:49:22.119550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:22.119523 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:22.175295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:22.175246 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" podStartSLOduration=3.175194533 podStartE2EDuration="3.175194533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:22.172686281 +0000 UTC m=+2232.293739126" watchObservedRunningTime="2026-11-03 16:49:22.175194533 +0000 UTC m=+2232.296247410" Nov 03 16:49:22.944294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:22.944262 5946 generic.go:356] "Generic (PLEG): container finished" podID="04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" containerID="edde3b0db4c09d8eb38f03ccb2eaa2b2271ccd4acbc95ea89728e61f17f4323f" exitCode=0 Nov 03 16:49:22.944764 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:22.944357 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" event={"ID":"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6","Type":"ContainerDied","Data":"edde3b0db4c09d8eb38f03ccb2eaa2b2271ccd4acbc95ea89728e61f17f4323f"} Nov 03 16:49:22.946021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:22.945992 5946 generic.go:356] "Generic (PLEG): container finished" podID="e093780f-d756-4959-b202-b772c21aed17" containerID="054c6762db8266993ab40a5d3fa80fc56ed6bab758783cdfff99bdb3d4c16581" exitCode=0 Nov 03 16:49:22.946133 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:22.946071 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" event={"ID":"e093780f-d756-4959-b202-b772c21aed17","Type":"ContainerDied","Data":"054c6762db8266993ab40a5d3fa80fc56ed6bab758783cdfff99bdb3d4c16581"} Nov 03 16:49:23.114352 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.114326 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:23.136396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.136324 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-openshift-ca\") pod \"7cede107-03a3-40fe-b87f-2b91928d568f\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " Nov 03 16:49:23.136531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.136483 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cede107-03a3-40fe-b87f-2b91928d568f-kube-api-access-dqxkv\" (UniqueName: \"kubernetes.io/projected/7cede107-03a3-40fe-b87f-2b91928d568f-kube-api-access-dqxkv\") pod \"7cede107-03a3-40fe-b87f-2b91928d568f\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " Nov 03 16:49:23.136592 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.136534 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-xks-ca\") pod \"7cede107-03a3-40fe-b87f-2b91928d568f\" (UID: \"7cede107-03a3-40fe-b87f-2b91928d568f\") " Nov 03 16:49:23.136707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.136685 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-openshift-ca" pod "7cede107-03a3-40fe-b87f-2b91928d568f" (UID: "7cede107-03a3-40fe-b87f-2b91928d568f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:23.136990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.136973 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:23.138659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.138633 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-xks-ca" pod "7cede107-03a3-40fe-b87f-2b91928d568f" (UID: "7cede107-03a3-40fe-b87f-2b91928d568f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:23.138766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.138678 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cede107-03a3-40fe-b87f-2b91928d568f-kube-api-access-dqxkv" pod "7cede107-03a3-40fe-b87f-2b91928d568f" (UID: "7cede107-03a3-40fe-b87f-2b91928d568f"). InnerVolumeSpecName "kube-api-access-dqxkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:23.238322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.238291 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxkv\" (UniqueName: \"kubernetes.io/projected/7cede107-03a3-40fe-b87f-2b91928d568f-kube-api-access-dqxkv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:23.238322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.238318 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cede107-03a3-40fe-b87f-2b91928d568f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:23.956157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.956123 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" Nov 03 16:49:23.956604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.956165 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk" event={"ID":"7cede107-03a3-40fe-b87f-2b91928d568f","Type":"ContainerDied","Data":"e19b91dbf452a010d3db2d8091ec953dda54dac192a03e1939bb150514eb5866"} Nov 03 16:49:23.956604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:23.956293 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e19b91dbf452a010d3db2d8091ec953dda54dac192a03e1939bb150514eb5866" Nov 03 16:49:24.133187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.133159 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:24.148997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.148970 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e093780f-d756-4959-b202-b772c21aed17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093780f-d756-4959-b202-b772c21aed17-maas-api-xks-ca\") pod \"e093780f-d756-4959-b202-b772c21aed17\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " Nov 03 16:49:24.149113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.149036 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e093780f-d756-4959-b202-b772c21aed17-kube-api-access-5cgzn\" (UniqueName: \"kubernetes.io/projected/e093780f-d756-4959-b202-b772c21aed17-kube-api-access-5cgzn\") pod \"e093780f-d756-4959-b202-b772c21aed17\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " Nov 03 16:49:24.149326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.149305 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e093780f-d756-4959-b202-b772c21aed17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093780f-d756-4959-b202-b772c21aed17-maas-api-openshift-ca\") pod \"e093780f-d756-4959-b202-b772c21aed17\" (UID: \"e093780f-d756-4959-b202-b772c21aed17\") " Nov 03 16:49:24.149721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.149693 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e093780f-d756-4959-b202-b772c21aed17-maas-api-openshift-ca" pod "e093780f-d756-4959-b202-b772c21aed17" (UID: "e093780f-d756-4959-b202-b772c21aed17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:24.149930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.149907 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093780f-d756-4959-b202-b772c21aed17-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:24.151142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.151120 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e093780f-d756-4959-b202-b772c21aed17-maas-api-xks-ca" pod "e093780f-d756-4959-b202-b772c21aed17" (UID: "e093780f-d756-4959-b202-b772c21aed17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:24.151307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.151288 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e093780f-d756-4959-b202-b772c21aed17-kube-api-access-5cgzn" pod "e093780f-d756-4959-b202-b772c21aed17" (UID: "e093780f-d756-4959-b202-b772c21aed17"). InnerVolumeSpecName "kube-api-access-5cgzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:24.173426 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.173406 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:24.251425 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.251399 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-openshift-ca\") pod \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " Nov 03 16:49:24.251603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.251471 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-kube-api-access-czrkg\" (UniqueName: \"kubernetes.io/projected/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-kube-api-access-czrkg\") pod \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " Nov 03 16:49:24.251603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.251507 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-xks-ca\") pod \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\" (UID: \"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6\") " Nov 03 16:49:24.251738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.251703 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-openshift-ca" pod "04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" (UID: "04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:24.252461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.251989 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093780f-d756-4959-b202-b772c21aed17-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:24.252461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.252014 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:24.252461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.252029 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cgzn\" (UniqueName: \"kubernetes.io/projected/e093780f-d756-4959-b202-b772c21aed17-kube-api-access-5cgzn\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:24.254685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.254646 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-xks-ca" pod "04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" (UID: "04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:24.254937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.254916 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-kube-api-access-czrkg" pod "04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" (UID: "04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6"). InnerVolumeSpecName "kube-api-access-czrkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:24.353086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.353052 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czrkg\" (UniqueName: \"kubernetes.io/projected/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-kube-api-access-czrkg\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:24.353086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.353082 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:24.965960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.965931 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" Nov 03 16:49:24.966560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.965964 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl" event={"ID":"e093780f-d756-4959-b202-b772c21aed17","Type":"ContainerDied","Data":"24fc06637ce51c90025f2c522576a92a9921f07c099970e3870ee71b59c15a0a"} Nov 03 16:49:24.966560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.966006 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24fc06637ce51c90025f2c522576a92a9921f07c099970e3870ee71b59c15a0a" Nov 03 16:49:24.967686 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.967637 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" event={"ID":"04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6","Type":"ContainerDied","Data":"984a5a17f9b3c9fc561e3ea25970c10d279f63ac4a913e7255d6e9d0a7bbcff0"} Nov 03 16:49:24.967845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.967692 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="984a5a17f9b3c9fc561e3ea25970c10d279f63ac4a913e7255d6e9d0a7bbcff0" Nov 03 16:49:24.967845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:24.967649 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c" Nov 03 16:49:26.403155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:26.403123 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:28.041917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:28.041877 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:28.308620 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:28.308522 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:28.491365 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:28.491336 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k"] Nov 03 16:49:28.502483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:28.502455 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:28.541787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:28.541753 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bb620-8tr2k"] Nov 03 16:49:28.931870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:28.931840 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:30.005833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.005800 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411c49f0-8316-415a-af3e-b97b8a250e12" path="/var/lib/kubelet/pods/411c49f0-8316-415a-af3e-b97b8a250e12/volumes" Nov 03 16:49:30.182451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.182422 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:30.770013 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.769977 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88"] Nov 03 16:49:30.770647 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770631 5946 cpu_manager.go:414] "containerMap: removing container" podUID="04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" containerName="revoke-keys" Nov 03 16:49:30.770715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770649 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" containerName="revoke-keys" Nov 03 16:49:30.770715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770669 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e093780f-d756-4959-b202-b772c21aed17" containerName="revoke-keys" Nov 03 16:49:30.770715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770674 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e093780f-d756-4959-b202-b772c21aed17" containerName="revoke-keys" Nov 03 16:49:30.770715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770688 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7cede107-03a3-40fe-b87f-2b91928d568f" containerName="revoke-keys" Nov 03 16:49:30.770715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770694 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cede107-03a3-40fe-b87f-2b91928d568f" containerName="revoke-keys" Nov 03 16:49:30.770901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770774 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e093780f-d756-4959-b202-b772c21aed17" containerName="revoke-keys" Nov 03 16:49:30.770901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770784 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cede107-03a3-40fe-b87f-2b91928d568f" containerName="revoke-keys" Nov 03 16:49:30.770901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.770798 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" containerName="revoke-keys" Nov 03 16:49:30.774475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.774458 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.833451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.833415 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88"] Nov 03 16:49:30.848545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.848513 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.848712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.848581 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhb9h\" (UniqueName: \"kubernetes.io/projected/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-kube-api-access-bhb9h\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.848712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.848685 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.880808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.880700 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:49:30.881129 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.881110 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:49:30.881350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.881168 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:49:30.881533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.881302 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:49:30.949819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.949784 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.950004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.949931 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.950004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.949976 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhb9h\" (UniqueName: \"kubernetes.io/projected/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-kube-api-access-bhb9h\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.950712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.950634 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:30.952373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:30.952352 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:31.010943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:31.010901 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhb9h\" (UniqueName: \"kubernetes.io/projected/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-kube-api-access-bhb9h\") pod \"maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:31.088854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:31.088775 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:31.480421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:31.480386 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88"] Nov 03 16:49:31.817854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:31.817827 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl"] Nov 03 16:49:32.038420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:32.038378 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" event={"ID":"6f8a06e3-f2b2-4e3f-9991-7aba2f096636","Type":"ContainerStarted","Data":"73e5d1f1bf36fbf4e8c6b07a2b5531378aee13190f45bd2c6b1bd213f46d255a"} Nov 03 16:49:32.038868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:32.038448 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" event={"ID":"6f8a06e3-f2b2-4e3f-9991-7aba2f096636","Type":"ContainerStarted","Data":"de23d06218e7bbe7eda900cc6b5cd74dadcfd544f4224c86798b7c125d374062"} Nov 03 16:49:32.112732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:32.112648 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:32.162674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:32.162604 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" podStartSLOduration=2.162588689 podStartE2EDuration="2.162588689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:32.15967679 +0000 UTC m=+2242.280729612" watchObservedRunningTime="2026-11-03 16:49:32.162588689 +0000 UTC m=+2242.283641512" Nov 03 16:49:32.293960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:32.293929 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:32.849476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:32.849388 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59"] Nov 03 16:49:32.849697 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:32.849641 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" containerID="cri-o://bc2a74134eccf40d504b9105cc4d0ec36bd2d766aba42586bed953b0dd299b9e" gracePeriod=30 Nov 03 16:49:33.047635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:33.047599 5946 generic.go:356] "Generic (PLEG): container finished" podID="6f8a06e3-f2b2-4e3f-9991-7aba2f096636" containerID="73e5d1f1bf36fbf4e8c6b07a2b5531378aee13190f45bd2c6b1bd213f46d255a" exitCode=0 Nov 03 16:49:33.048041 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:33.047684 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" event={"ID":"6f8a06e3-f2b2-4e3f-9991-7aba2f096636","Type":"ContainerDied","Data":"73e5d1f1bf36fbf4e8c6b07a2b5531378aee13190f45bd2c6b1bd213f46d255a"} Nov 03 16:49:33.498142 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:33.498108 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:34.203124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.203097 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:34.296011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.295974 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-kube-api-access-bhb9h\" (UniqueName: \"kubernetes.io/projected/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-kube-api-access-bhb9h\") pod \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " Nov 03 16:49:34.296259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.296037 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-openshift-ca\") pod \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " Nov 03 16:49:34.296259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.296192 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-xks-ca\") pod \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\" (UID: \"6f8a06e3-f2b2-4e3f-9991-7aba2f096636\") " Nov 03 16:49:34.296462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.296436 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-openshift-ca" pod "6f8a06e3-f2b2-4e3f-9991-7aba2f096636" (UID: "6f8a06e3-f2b2-4e3f-9991-7aba2f096636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:34.296840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.296815 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:34.298611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.298578 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-kube-api-access-bhb9h" pod "6f8a06e3-f2b2-4e3f-9991-7aba2f096636" (UID: "6f8a06e3-f2b2-4e3f-9991-7aba2f096636"). InnerVolumeSpecName "kube-api-access-bhb9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:34.298741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.298724 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-xks-ca" pod "6f8a06e3-f2b2-4e3f-9991-7aba2f096636" (UID: "6f8a06e3-f2b2-4e3f-9991-7aba2f096636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:34.309107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.309080 5946 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:49:34.309239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.309132 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:49:34.398899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.398863 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:34.398899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.398895 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhb9h\" (UniqueName: \"kubernetes.io/projected/6f8a06e3-f2b2-4e3f-9991-7aba2f096636-kube-api-access-bhb9h\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:34.667182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:34.666904 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:49:35.063294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:35.063253 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" event={"ID":"6f8a06e3-f2b2-4e3f-9991-7aba2f096636","Type":"ContainerDied","Data":"de23d06218e7bbe7eda900cc6b5cd74dadcfd544f4224c86798b7c125d374062"} Nov 03 16:49:35.063502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:35.063312 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de23d06218e7bbe7eda900cc6b5cd74dadcfd544f4224c86798b7c125d374062" Nov 03 16:49:35.063502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:35.063272 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88" Nov 03 16:49:35.136451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:35.136421 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp"] Nov 03 16:49:35.328315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:35.328170 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:35.550736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:35.550707 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm"] Nov 03 16:49:35.985254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:35.981391 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:36.178970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:36.178933 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:36.818873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:36.818809 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" podUID="a1665afc-3f47-47e5-854d-051b26ba4782" containerName="payload-pre-processing" containerID="cri-o://54204414457e1532274c571ab1963ee405971e34d712eb7503f7481c207fd656" gracePeriod=25 Nov 03 16:49:36.950413 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:36.950380 5946 logging.go:55] [core] [Channel #2657 SubChannel #2658]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.167:50052", ServerName: "10.134.0.167:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.167:50052: connect: connection refused" Nov 03 16:49:37.079145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.079111 5946 generic.go:356] "Generic (PLEG): container finished" podID="a1665afc-3f47-47e5-854d-051b26ba4782" containerID="54204414457e1532274c571ab1963ee405971e34d712eb7503f7481c207fd656" exitCode=0 Nov 03 16:49:37.079323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.079159 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" event={"ID":"a1665afc-3f47-47e5-854d-051b26ba4782","Type":"ContainerDied","Data":"54204414457e1532274c571ab1963ee405971e34d712eb7503f7481c207fd656"} Nov 03 16:49:37.079323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.079261 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" event={"ID":"a1665afc-3f47-47e5-854d-051b26ba4782","Type":"ContainerDied","Data":"1575922eb8f2de55cb528cf3ebc23dfd013ef012acd4a21234030036756f7bd6"} Nov 03 16:49:37.079323 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.079282 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1575922eb8f2de55cb528cf3ebc23dfd013ef012acd4a21234030036756f7bd6" Nov 03 16:49:37.090253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.090226 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:49:37.184232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.183776 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:37.243645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.243604 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1665afc-3f47-47e5-854d-051b26ba4782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1665afc-3f47-47e5-854d-051b26ba4782-plugins-config-volume\") pod \"a1665afc-3f47-47e5-854d-051b26ba4782\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " Nov 03 16:49:37.243862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.243795 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1665afc-3f47-47e5-854d-051b26ba4782-kube-api-access-7w64w\" (UniqueName: \"kubernetes.io/projected/a1665afc-3f47-47e5-854d-051b26ba4782-kube-api-access-7w64w\") pod \"a1665afc-3f47-47e5-854d-051b26ba4782\" (UID: \"a1665afc-3f47-47e5-854d-051b26ba4782\") " Nov 03 16:49:37.247298 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.246709 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1665afc-3f47-47e5-854d-051b26ba4782-plugins-config-volume" pod "a1665afc-3f47-47e5-854d-051b26ba4782" (UID: "a1665afc-3f47-47e5-854d-051b26ba4782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:37.248317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.248271 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1665afc-3f47-47e5-854d-051b26ba4782-kube-api-access-7w64w" pod "a1665afc-3f47-47e5-854d-051b26ba4782" (UID: "a1665afc-3f47-47e5-854d-051b26ba4782"). InnerVolumeSpecName "kube-api-access-7w64w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:37.345847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.345764 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1665afc-3f47-47e5-854d-051b26ba4782-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:37.345847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.345794 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w64w\" (UniqueName: \"kubernetes.io/projected/a1665afc-3f47-47e5-854d-051b26ba4782-kube-api-access-7w64w\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:37.444932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.444897 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:37.639266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.639148 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:37.739823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.739781 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8"] Nov 03 16:49:37.740266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.740237 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" containerID="cri-o://c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d" gracePeriod=30 Nov 03 16:49:37.775431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.775395 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4"] Nov 03 16:49:37.775938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.775894 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" containerID="cri-o://84dd1ac35b0c3dd40e5ed9e0c4e3a16a580e3ceea5c0847054a4e30a6ea55de8" gracePeriod=30 Nov 03 16:49:37.884988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.884942 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:37.950926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:37.950881 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" podUID="a1665afc-3f47-47e5-854d-051b26ba4782" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.167:50052\" within 1s: context deadline exceeded" Nov 03 16:49:37.951087 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:37.951000 5946 logging.go:55] [core] [Channel #2657 SubChannel #2658]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.167:50052", ServerName: "10.134.0.167:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.167:50052: operation was canceled" Nov 03 16:49:38.088888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.088849 5946 generic.go:356] "Generic (PLEG): container finished" podID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerID="bc2a74134eccf40d504b9105cc4d0ec36bd2d766aba42586bed953b0dd299b9e" exitCode=0 Nov 03 16:49:38.089070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.088912 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" event={"ID":"b892a281-9708-45e3-b454-ce3d9abfd2cd","Type":"ContainerDied","Data":"bc2a74134eccf40d504b9105cc4d0ec36bd2d766aba42586bed953b0dd299b9e"} Nov 03 16:49:38.089119 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.089070 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl" Nov 03 16:49:38.267906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.267862 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl"] Nov 03 16:49:38.273017 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.272982 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66f145-7655c65cf-rmjxl"] Nov 03 16:49:38.559031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.559002 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:49:38.592136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592105 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-kube-api-access-4zkjb\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-kube-api-access-4zkjb\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592147 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b892a281-9708-45e3-b454-ce3d9abfd2cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b892a281-9708-45e3-b454-ce3d9abfd2cd-istiod-ca-cert\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592182 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-token\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-token\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592238 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-socket\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592265 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-certs\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592298 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-data\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592339 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-credential-socket\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592557 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-podinfo\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592589 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592569 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-socket" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:38.592764 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592591 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b892a281-9708-45e3-b454-ce3d9abfd2cd-istiod-ca-cert" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:38.592764 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592753 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-data" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:38.592878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592805 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-envoy\") pod \"b892a281-9708-45e3-b454-ce3d9abfd2cd\" (UID: \"b892a281-9708-45e3-b454-ce3d9abfd2cd\") " Nov 03 16:49:38.592981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.592960 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-credential-socket" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:38.594683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.593046 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-certs" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:38.594683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.593385 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b892a281-9708-45e3-b454-ce3d9abfd2cd-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.594683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.593404 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.594683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.593417 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.594683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.593425 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.594683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.593434 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.595440 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.595413 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-token" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:38.595565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.595548 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-envoy" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:38.595635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.595564 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-podinfo" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:38.595734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.595717 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-kube-api-access-4zkjb" pod "b892a281-9708-45e3-b454-ce3d9abfd2cd" (UID: "b892a281-9708-45e3-b454-ce3d9abfd2cd"). InnerVolumeSpecName "kube-api-access-4zkjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:38.694837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.694803 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.694837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.694835 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.695047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.694847 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkjb\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-kube-api-access-4zkjb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.695047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.694860 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b892a281-9708-45e3-b454-ce3d9abfd2cd-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:38.704974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:38.704930 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:39.033681 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:39.033643 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:49:39.034093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:39.033716 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:49:39.102527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:39.102484 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" event={"ID":"b892a281-9708-45e3-b454-ce3d9abfd2cd","Type":"ContainerDied","Data":"523681b3b7754a577f324642be40a80c5390e25fdb98b1298acbc89039047b78"} Nov 03 16:49:39.102527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:39.102512 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59" Nov 03 16:49:39.102858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:39.102562 5946 scope.go:122] "RemoveContainer" containerID="bc2a74134eccf40d504b9105cc4d0ec36bd2d766aba42586bed953b0dd299b9e" Nov 03 16:49:39.275431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:39.275396 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59"] Nov 03 16:49:39.326292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:39.326217 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-2xj59"] Nov 03 16:49:40.004505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.004472 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1665afc-3f47-47e5-854d-051b26ba4782" path="/var/lib/kubelet/pods/a1665afc-3f47-47e5-854d-051b26ba4782/volumes" Nov 03 16:49:40.005137 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.005120 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" path="/var/lib/kubelet/pods/b892a281-9708-45e3-b454-ce3d9abfd2cd/volumes" Nov 03 16:49:40.196597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.196562 5946 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:49:40.196988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.196616 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:49:40.551863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.551816 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" podUID="f8cf0120-b157-4ffc-82be-ff5bc16776d4" containerName="payload-processing" containerID="cri-o://a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4" gracePeriod=25 Nov 03 16:49:40.820810 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.820729 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:49:40.925418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.925383 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cf0120-b157-4ffc-82be-ff5bc16776d4-kube-api-access-js9cl\" (UniqueName: \"kubernetes.io/projected/f8cf0120-b157-4ffc-82be-ff5bc16776d4-kube-api-access-js9cl\") pod \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " Nov 03 16:49:40.925613 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.925594 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cf0120-b157-4ffc-82be-ff5bc16776d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cf0120-b157-4ffc-82be-ff5bc16776d4-plugins-config-volume\") pod \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\" (UID: \"f8cf0120-b157-4ffc-82be-ff5bc16776d4\") " Nov 03 16:49:40.925956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.925932 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cf0120-b157-4ffc-82be-ff5bc16776d4-plugins-config-volume" pod "f8cf0120-b157-4ffc-82be-ff5bc16776d4" (UID: "f8cf0120-b157-4ffc-82be-ff5bc16776d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:40.926232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.926097 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cf0120-b157-4ffc-82be-ff5bc16776d4-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:40.927689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:40.927667 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cf0120-b157-4ffc-82be-ff5bc16776d4-kube-api-access-js9cl" pod "f8cf0120-b157-4ffc-82be-ff5bc16776d4" (UID: "f8cf0120-b157-4ffc-82be-ff5bc16776d4"). InnerVolumeSpecName "kube-api-access-js9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:41.027721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.027688 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9cl\" (UniqueName: \"kubernetes.io/projected/f8cf0120-b157-4ffc-82be-ff5bc16776d4-kube-api-access-js9cl\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:41.118092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.118005 5946 generic.go:356] "Generic (PLEG): container finished" podID="f8cf0120-b157-4ffc-82be-ff5bc16776d4" containerID="a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4" exitCode=0 Nov 03 16:49:41.118092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.118048 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" Nov 03 16:49:41.118314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.118098 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" event={"ID":"f8cf0120-b157-4ffc-82be-ff5bc16776d4","Type":"ContainerDied","Data":"a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4"} Nov 03 16:49:41.118314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.118150 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm" event={"ID":"f8cf0120-b157-4ffc-82be-ff5bc16776d4","Type":"ContainerDied","Data":"5afd7512528d060244b0297d1d5aa448fa58786853bf2dab749b085a473c68f1"} Nov 03 16:49:41.118314 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.118175 5946 scope.go:122] "RemoveContainer" containerID="a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4" Nov 03 16:49:41.129551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.129534 5946 scope.go:122] "RemoveContainer" containerID="a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4" Nov 03 16:49:41.129833 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:49:41.129814 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4\": container with ID starting with a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4 not found: ID does not exist" containerID="a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4" Nov 03 16:49:41.129894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.129841 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4"} err="failed to get container status \"a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4\": rpc error: code = NotFound desc = could not find container \"a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4\": container with ID starting with a4da50072b7c24e5ab0fe7502b2e3a4350f8ba4254e7647a5ee88c08546bead4 not found: ID does not exist" Nov 03 16:49:41.290798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.290753 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm"] Nov 03 16:49:41.296124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:41.296093 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c626cd-db85cd65f-vmpfm"] Nov 03 16:49:42.005175 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:42.005141 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8cf0120-b157-4ffc-82be-ff5bc16776d4" path="/var/lib/kubelet/pods/f8cf0120-b157-4ffc-82be-ff5bc16776d4/volumes" Nov 03 16:49:42.316921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:42.316846 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4"] Nov 03 16:49:42.429381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:42.429344 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj"] Nov 03 16:49:42.586366 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:49:42.586277 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:49:42.715820 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:42.715782 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:42.907936 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:42.907901 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:43.099270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.097112 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:43.110317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.110242 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:49:43.138053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.138014 5946 generic.go:356] "Generic (PLEG): container finished" podID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerID="c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d" exitCode=0 Nov 03 16:49:43.138247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.138131 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" event={"ID":"924e9362-8f6a-4bfa-b450-2427d1c22aa8","Type":"ContainerDied","Data":"c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d"} Nov 03 16:49:43.138305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.138247 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" event={"ID":"924e9362-8f6a-4bfa-b450-2427d1c22aa8","Type":"ContainerDied","Data":"7dd3dd030b92f407157590e51189db4d88d3fa9cf7db442c901ac0a4f597535a"} Nov 03 16:49:43.138305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.138278 5946 scope.go:122] "RemoveContainer" containerID="c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d" Nov 03 16:49:43.138437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.138423 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8" Nov 03 16:49:43.140588 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.140558 5946 generic.go:356] "Generic (PLEG): container finished" podID="b764326c-bef1-4e49-bb24-e53f23b38526" containerID="84dd1ac35b0c3dd40e5ed9e0c4e3a16a580e3ceea5c0847054a4e30a6ea55de8" exitCode=0 Nov 03 16:49:43.140758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.140719 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" event={"ID":"b764326c-bef1-4e49-bb24-e53f23b38526","Type":"ContainerDied","Data":"84dd1ac35b0c3dd40e5ed9e0c4e3a16a580e3ceea5c0847054a4e30a6ea55de8"} Nov 03 16:49:43.140878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.140780 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" event={"ID":"b764326c-bef1-4e49-bb24-e53f23b38526","Type":"ContainerDied","Data":"6e4df9c9c9c560e8016878bec25c57b4d4d5dde0fe7bc862e677ad40d1838920"} Nov 03 16:49:43.140878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.140806 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e4df9c9c9c560e8016878bec25c57b4d4d5dde0fe7bc862e677ad40d1838920" Nov 03 16:49:43.150272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.150235 5946 scope.go:122] "RemoveContainer" containerID="c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d" Nov 03 16:49:43.150597 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:49:43.150569 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d\": container with ID starting with c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d not found: ID does not exist" containerID="c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d" Nov 03 16:49:43.150698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.150608 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d"} err="failed to get container status \"c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d\": rpc error: code = NotFound desc = could not find container \"c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d\": container with ID starting with c076a1bbec7c70f5dcfb3def0d3b6fa35a4b03276fa55e537d3b0eabbf74a43d not found: ID does not exist" Nov 03 16:49:43.151220 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.151187 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:49:43.158240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158098 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-credential-socket\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158182 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-certs\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158236 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-token\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-token\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158287 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-data\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158349 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-socket\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158393 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-envoy\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158429 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-podinfo\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158458 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-kube-api-access-wckpd\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-kube-api-access-wckpd\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.158692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.158610 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istiod-ca-cert\") pod \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\" (UID: \"924e9362-8f6a-4bfa-b450-2427d1c22aa8\") " Nov 03 16:49:43.164641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.159101 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-credential-socket" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.164641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.159380 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-socket" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.164641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.159554 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-certs" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.164641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.160366 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-data" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.180282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.173776 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.180282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.173807 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.180282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.173821 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.180282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.173836 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.180282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.174021 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istiod-ca-cert" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:43.200752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.200697 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-token" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:43.201773 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.201748 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-podinfo" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:43.228421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.227918 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-kube-api-access-wckpd" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "kube-api-access-wckpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:43.228421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.228094 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-envoy" pod "924e9362-8f6a-4bfa-b450-2427d1c22aa8" (UID: "924e9362-8f6a-4bfa-b450-2427d1c22aa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.274733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.274702 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-data\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.274921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.274855 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-kube-api-access-lpwk9\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-kube-api-access-lpwk9\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.274921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.274894 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b764326c-bef1-4e49-bb24-e53f23b38526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b764326c-bef1-4e49-bb24-e53f23b38526-istio-podinfo\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.275072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275040 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-data" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.275165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275080 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-socket\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.275265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275160 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-certs\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.275390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275363 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-socket" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.275390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275376 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-envoy\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.275528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275414 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-istio-token\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-istio-token\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.275528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275452 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-credential-socket\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.275528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275491 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b764326c-bef1-4e49-bb24-e53f23b38526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b764326c-bef1-4e49-bb24-e53f23b38526-istiod-ca-cert\") pod \"b764326c-bef1-4e49-bb24-e53f23b38526\" (UID: \"b764326c-bef1-4e49-bb24-e53f23b38526\") " Nov 03 16:49:43.275790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.275766 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-certs" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.276147 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276123 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276162 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276178 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276191 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276222 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276237 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wckpd\" (UniqueName: \"kubernetes.io/projected/924e9362-8f6a-4bfa-b450-2427d1c22aa8-kube-api-access-wckpd\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276252 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276266 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924e9362-8f6a-4bfa-b450-2427d1c22aa8-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.276655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276129 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b764326c-bef1-4e49-bb24-e53f23b38526-istiod-ca-cert" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:43.276655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.276341 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-credential-socket" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.278586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.278554 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-istio-token" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:43.278866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.278654 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b764326c-bef1-4e49-bb24-e53f23b38526-istio-podinfo" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:43.278866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.278712 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-kube-api-access-lpwk9" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "kube-api-access-lpwk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:43.280195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.280170 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-envoy" pod "b764326c-bef1-4e49-bb24-e53f23b38526" (UID: "b764326c-bef1-4e49-bb24-e53f23b38526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:43.377888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.377809 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.377888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.377838 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.377888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.377847 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b764326c-bef1-4e49-bb24-e53f23b38526-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.377888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.377856 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b764326c-bef1-4e49-bb24-e53f23b38526-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.377888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.377866 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwk9\" (UniqueName: \"kubernetes.io/projected/b764326c-bef1-4e49-bb24-e53f23b38526-kube-api-access-lpwk9\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.377888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.377876 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b764326c-bef1-4e49-bb24-e53f23b38526-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:43.566214 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.566163 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8"] Nov 03 16:49:43.572801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:43.572772 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-gfhn8"] Nov 03 16:49:44.005273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.005242 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" path="/var/lib/kubelet/pods/924e9362-8f6a-4bfa-b450-2427d1c22aa8/volumes" Nov 03 16:49:44.163231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.159431 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4" Nov 03 16:49:44.337685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.337596 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4"] Nov 03 16:49:44.388464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.388428 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-9c5k4"] Nov 03 16:49:44.442545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.442503 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:44.735975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.735946 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:44.942417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.942381 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r"] Nov 03 16:49:44.950449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.950423 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0ea8a5-24w6r"] Nov 03 16:49:44.953296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:44.953260 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:45.349590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:45.349563 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:46.013259 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:46.013222 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" path="/var/lib/kubelet/pods/b764326c-bef1-4e49-bb24-e53f23b38526/volumes" Nov 03 16:49:46.013663 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:46.013650 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ae9b93-97a5-498c-8189-dff362071fce" path="/var/lib/kubelet/pods/f6ae9b93-97a5-498c-8189-dff362071fce/volumes" Nov 03 16:49:47.317524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:47.317479 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" podUID="5078901f-7793-4090-8611-36726bb133c7" containerName="payload-processing" containerID="cri-o://a9053269f29c18093e25910de20daf304aaed92e383433297d9aad9a22aa1c5d" gracePeriod=25 Nov 03 16:49:47.430570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:47.430530 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" podUID="aad44a8d-f155-4329-afd7-ceaed3489ede" containerName="payload-pre-processing" containerID="cri-o://71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e" gracePeriod=25 Nov 03 16:49:49.174368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:49.174287 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:50.158253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.158185 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" containerID="cri-o://c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289" gracePeriod=45 Nov 03 16:49:50.225077 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.225043 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused" Nov 03 16:49:50.427179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.427113 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:49:50.576527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.576496 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7843122b-449a-4188-8755-b49147f6f8ed-kube-api-access-tbpkf\" (UniqueName: \"kubernetes.io/projected/7843122b-449a-4188-8755-b49147f6f8ed-kube-api-access-tbpkf\") pod \"7843122b-449a-4188-8755-b49147f6f8ed\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " Nov 03 16:49:50.576731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.576597 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-tmp-dir\") pod \"7843122b-449a-4188-8755-b49147f6f8ed\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " Nov 03 16:49:50.576731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.576703 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-dshm\") pod \"7843122b-449a-4188-8755-b49147f6f8ed\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " Nov 03 16:49:50.576892 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.576757 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-home\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-home\") pod \"7843122b-449a-4188-8755-b49147f6f8ed\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " Nov 03 16:49:50.577349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.576852 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-tmp-dir" pod "7843122b-449a-4188-8755-b49147f6f8ed" (UID: "7843122b-449a-4188-8755-b49147f6f8ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:50.577349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.577038 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-home" pod "7843122b-449a-4188-8755-b49147f6f8ed" (UID: "7843122b-449a-4188-8755-b49147f6f8ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:50.577734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.577353 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-model-cache\") pod \"7843122b-449a-4188-8755-b49147f6f8ed\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " Nov 03 16:49:50.577734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.577476 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7843122b-449a-4188-8755-b49147f6f8ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/7843122b-449a-4188-8755-b49147f6f8ed-tls-certs\") pod \"7843122b-449a-4188-8755-b49147f6f8ed\" (UID: \"7843122b-449a-4188-8755-b49147f6f8ed\") " Nov 03 16:49:50.577734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.577575 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-model-cache" pod "7843122b-449a-4188-8755-b49147f6f8ed" (UID: "7843122b-449a-4188-8755-b49147f6f8ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:50.578049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.578026 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:50.578225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.578192 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:50.578517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.578495 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:50.579442 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.579413 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-dshm" pod "7843122b-449a-4188-8755-b49147f6f8ed" (UID: "7843122b-449a-4188-8755-b49147f6f8ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:50.579561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.579508 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7843122b-449a-4188-8755-b49147f6f8ed-kube-api-access-tbpkf" pod "7843122b-449a-4188-8755-b49147f6f8ed" (UID: "7843122b-449a-4188-8755-b49147f6f8ed"). InnerVolumeSpecName "kube-api-access-tbpkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:50.579837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.579811 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7843122b-449a-4188-8755-b49147f6f8ed-tls-certs" pod "7843122b-449a-4188-8755-b49147f6f8ed" (UID: "7843122b-449a-4188-8755-b49147f6f8ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:50.679614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.679533 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbpkf\" (UniqueName: \"kubernetes.io/projected/7843122b-449a-4188-8755-b49147f6f8ed-kube-api-access-tbpkf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:50.679614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.679564 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843122b-449a-4188-8755-b49147f6f8ed-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:50.679614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.679576 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7843122b-449a-4188-8755-b49147f6f8ed-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:50.863364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:50.863315 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:51.141851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.141816 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:51.212946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.212914 5946 generic.go:356] "Generic (PLEG): container finished" podID="7843122b-449a-4188-8755-b49147f6f8ed" containerID="c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289" exitCode=0 Nov 03 16:49:51.213130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.212971 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" Nov 03 16:49:51.213130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.212983 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" event={"ID":"7843122b-449a-4188-8755-b49147f6f8ed","Type":"ContainerDied","Data":"c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289"} Nov 03 16:49:51.213130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.213018 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp" event={"ID":"7843122b-449a-4188-8755-b49147f6f8ed","Type":"ContainerDied","Data":"9e161db45ff83fc3b766d51907e85a23850a689e2ee5b1fb9dae397e977a3380"} Nov 03 16:49:51.213130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.213035 5946 scope.go:122] "RemoveContainer" containerID="c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289" Nov 03 16:49:51.224903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.224872 5946 scope.go:122] "RemoveContainer" containerID="c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289" Nov 03 16:49:51.225220 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:49:51.225179 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289\": container with ID starting with c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289 not found: ID does not exist" containerID="c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289" Nov 03 16:49:51.225545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.225281 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289"} err="failed to get container status \"c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289\": rpc error: code = NotFound desc = could not find container \"c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289\": container with ID starting with c51c44f83c100a853890143854d592396d2727934713d491bbdeebdeaf142289 not found: ID does not exist" Nov 03 16:49:51.306361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.306325 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh"] Nov 03 16:49:51.321622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.321586 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0fda73-bwclh"] Nov 03 16:49:51.395551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.395469 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp"] Nov 03 16:49:51.405720 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.405689 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/simulator-w6-547a38-kserve-6b9c556555-c2gsp"] Nov 03 16:49:51.408715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.408677 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:51.853520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:51.853464 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:52.004436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:52.004403 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cae2190-3e4c-4cb5-8c31-46c89eae2d3f" path="/var/lib/kubelet/pods/6cae2190-3e4c-4cb5-8c31-46c89eae2d3f/volumes" Nov 03 16:49:52.004849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:52.004836 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7843122b-449a-4188-8755-b49147f6f8ed" path="/var/lib/kubelet/pods/7843122b-449a-4188-8755-b49147f6f8ed/volumes" Nov 03 16:49:52.066819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:52.066777 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" podUID="5078901f-7793-4090-8611-36726bb133c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:49:52.066978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:52.066783 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" podUID="aad44a8d-f155-4329-afd7-ceaed3489ede" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:49:54.135169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:54.135104 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:54.365328 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:54.365299 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:54.498742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:54.498707 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6"] Nov 03 16:49:54.498960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:54.498933 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" containerID="cri-o://61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3" gracePeriod=30 Nov 03 16:49:54.657076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:54.657037 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:55.022178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.022149 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:55.491793 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.491763 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh"] Nov 03 16:49:55.492326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492311 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" Nov 03 16:49:55.492326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492327 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492344 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492349 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492366 5946 cpu_manager.go:414] "containerMap: removing container" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492371 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492393 5946 cpu_manager.go:414] "containerMap: removing container" podUID="6f8a06e3-f2b2-4e3f-9991-7aba2f096636" containerName="revoke-keys" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492398 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8a06e3-f2b2-4e3f-9991-7aba2f096636" containerName="revoke-keys" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492407 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492412 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492418 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f8cf0120-b157-4ffc-82be-ff5bc16776d4" containerName="payload-processing" Nov 03 16:49:55.492421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492423 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cf0120-b157-4ffc-82be-ff5bc16776d4" containerName="payload-processing" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492429 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a1665afc-3f47-47e5-854d-051b26ba4782" containerName="payload-pre-processing" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492433 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1665afc-3f47-47e5-854d-051b26ba4782" containerName="payload-pre-processing" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492505 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8a06e3-f2b2-4e3f-9991-7aba2f096636" containerName="revoke-keys" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492513 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7843122b-449a-4188-8755-b49147f6f8ed" containerName="main" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492518 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cf0120-b157-4ffc-82be-ff5bc16776d4" containerName="payload-processing" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492525 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1665afc-3f47-47e5-854d-051b26ba4782" containerName="payload-pre-processing" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492532 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b764326c-bef1-4e49-bb24-e53f23b38526" containerName="istio-proxy" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492537 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b892a281-9708-45e3-b454-ce3d9abfd2cd" containerName="istio-proxy" Nov 03 16:49:55.492729 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.492544 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="924e9362-8f6a-4bfa-b450-2427d1c22aa8" containerName="istio-proxy" Nov 03 16:49:55.497390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.497367 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.555287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.555255 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:49:55.555798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.555757 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:49:55.562158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.562126 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh"] Nov 03 16:49:55.601690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.601655 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:49:55.601857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.601720 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:49:55.607066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.607040 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr"] Nov 03 16:49:55.611247 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.611224 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.639558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.639527 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfhc\" (UniqueName: \"kubernetes.io/projected/78eae67e-f178-496b-9a7d-f70f553dd5b4-kube-api-access-jwfhc\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.639696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.639569 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.639696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.639594 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.662042 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.662009 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr"] Nov 03 16:49:55.665465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.665436 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4"] Nov 03 16:49:55.670338 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.670312 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.728341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.728312 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4"] Nov 03 16:49:55.741459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.741417 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.741459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.741459 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqhpg\" (UniqueName: \"kubernetes.io/projected/bf063e91-8e8a-4ac6-a71c-52a8c333d953-kube-api-access-nqhpg\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.741714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.741613 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.741870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.741821 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfhc\" (UniqueName: \"kubernetes.io/projected/78eae67e-f178-496b-9a7d-f70f553dd5b4-kube-api-access-jwfhc\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.741945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.741865 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.741945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.741903 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.742837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.742815 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.744797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.744775 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.803072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.803037 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfhc\" (UniqueName: \"kubernetes.io/projected/78eae67e-f178-496b-9a7d-f70f553dd5b4-kube-api-access-jwfhc\") pod \"maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.810711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.810687 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:55.844472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.844436 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.844676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.844556 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwpg\" (UniqueName: \"kubernetes.io/projected/c0e0f140-4cdb-45a5-a934-83afee2ea73e-kube-api-access-7bwpg\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.844676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.844638 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.844676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.844663 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqhpg\" (UniqueName: \"kubernetes.io/projected/bf063e91-8e8a-4ac6-a71c-52a8c333d953-kube-api-access-nqhpg\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.844875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.844790 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.844875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.844828 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.845379 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.845356 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.847520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.847490 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.907130 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.907095 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqhpg\" (UniqueName: \"kubernetes.io/projected/bf063e91-8e8a-4ac6-a71c-52a8c333d953-kube-api-access-nqhpg\") pod \"maas-api-revoke-sub-8b23a481-k8ksr\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.925959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.925931 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:55.945961 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.945929 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwpg\" (UniqueName: \"kubernetes.io/projected/c0e0f140-4cdb-45a5-a934-83afee2ea73e-kube-api-access-7bwpg\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.946266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.946105 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.946266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.946186 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.947374 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.947347 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:55.948973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:55.948950 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:56.010667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:56.010583 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwpg\" (UniqueName: \"kubernetes.io/projected/c0e0f140-4cdb-45a5-a934-83afee2ea73e-kube-api-access-7bwpg\") pod \"maas-api-revoke-sub-65c2c3f8-6v6c4\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:56.199702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:56.199672 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh"] Nov 03 16:49:56.200944 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:56.200915 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78eae67e_f178_496b_9a7d_f70f553dd5b4.slice/crio-8f230e88458127ec26d49969670f289d905a2508d1e9ee35eefecd6c3b84c940 WatchSource:0}: Error finding container 8f230e88458127ec26d49969670f289d905a2508d1e9ee35eefecd6c3b84c940: Status 404 returned error can't find the container with id 8f230e88458127ec26d49969670f289d905a2508d1e9ee35eefecd6c3b84c940 Nov 03 16:49:56.251775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:56.251747 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" event={"ID":"78eae67e-f178-496b-9a7d-f70f553dd5b4","Type":"ContainerStarted","Data":"8f230e88458127ec26d49969670f289d905a2508d1e9ee35eefecd6c3b84c940"} Nov 03 16:49:56.288349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:56.288280 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:56.311610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:56.311578 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr"] Nov 03 16:49:56.313090 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:56.313063 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf063e91_8e8a_4ac6_a71c_52a8c333d953.slice/crio-2ebb845f14599be30d690737764221319cf41d63609544c924aa9ccc317e5dba WatchSource:0}: Error finding container 2ebb845f14599be30d690737764221319cf41d63609544c924aa9ccc317e5dba: Status 404 returned error can't find the container with id 2ebb845f14599be30d690737764221319cf41d63609544c924aa9ccc317e5dba Nov 03 16:49:56.598623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:56.598542 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:56.770578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:56.770553 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4"] Nov 03 16:49:56.772587 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:49:56.772562 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e0f140_4cdb_45a5_a934_83afee2ea73e.slice/crio-72092c22dfe7adde5d3ad95ccb9ae80190b6665c128f577ebb0cbc2dacc6fc2c WatchSource:0}: Error finding container 72092c22dfe7adde5d3ad95ccb9ae80190b6665c128f577ebb0cbc2dacc6fc2c: Status 404 returned error can't find the container with id 72092c22dfe7adde5d3ad95ccb9ae80190b6665c128f577ebb0cbc2dacc6fc2c Nov 03 16:49:57.260502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.260458 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" event={"ID":"c0e0f140-4cdb-45a5-a934-83afee2ea73e","Type":"ContainerStarted","Data":"6a63199083c54b220a32e1a286bf8c279734a7bfe57fdf69418591fb2d942e14"} Nov 03 16:49:57.260711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.260530 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" event={"ID":"c0e0f140-4cdb-45a5-a934-83afee2ea73e","Type":"ContainerStarted","Data":"72092c22dfe7adde5d3ad95ccb9ae80190b6665c128f577ebb0cbc2dacc6fc2c"} Nov 03 16:49:57.262124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.262091 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" event={"ID":"bf063e91-8e8a-4ac6-a71c-52a8c333d953","Type":"ContainerStarted","Data":"e5b5fb66c0382c00b3d0c7d15158ea5e7df485b0902b4343a8a3968296d83fb1"} Nov 03 16:49:57.262290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.262138 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" event={"ID":"bf063e91-8e8a-4ac6-a71c-52a8c333d953","Type":"ContainerStarted","Data":"2ebb845f14599be30d690737764221319cf41d63609544c924aa9ccc317e5dba"} Nov 03 16:49:57.263844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.263817 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" event={"ID":"78eae67e-f178-496b-9a7d-f70f553dd5b4","Type":"ContainerStarted","Data":"5246e2dc30f543c66d43a0bb8dab81a721d4eedeea0d52e3d181ad0829825ef9"} Nov 03 16:49:57.387925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.387867 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" podStartSLOduration=2.387852816 podStartE2EDuration="2.387852816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:57.386366485 +0000 UTC m=+2267.507419309" watchObservedRunningTime="2026-11-03 16:49:57.387852816 +0000 UTC m=+2267.508905640" Nov 03 16:49:57.511412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.511302 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" podStartSLOduration=2.511286632 podStartE2EDuration="2.511286632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:57.508822573 +0000 UTC m=+2267.629875410" watchObservedRunningTime="2026-11-03 16:49:57.511286632 +0000 UTC m=+2267.632339454" Nov 03 16:49:57.632188 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:57.632139 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" podStartSLOduration=2.6321264170000003 podStartE2EDuration="2.632126417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:57.630909772 +0000 UTC m=+2267.751962595" watchObservedRunningTime="2026-11-03 16:49:57.632126417 +0000 UTC m=+2267.753179240" Nov 03 16:49:58.272727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:58.272693 5946 generic.go:356] "Generic (PLEG): container finished" podID="78eae67e-f178-496b-9a7d-f70f553dd5b4" containerID="5246e2dc30f543c66d43a0bb8dab81a721d4eedeea0d52e3d181ad0829825ef9" exitCode=0 Nov 03 16:49:58.272911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:58.272778 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" event={"ID":"78eae67e-f178-496b-9a7d-f70f553dd5b4","Type":"ContainerDied","Data":"5246e2dc30f543c66d43a0bb8dab81a721d4eedeea0d52e3d181ad0829825ef9"} Nov 03 16:49:58.274581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:58.274555 5946 generic.go:356] "Generic (PLEG): container finished" podID="c0e0f140-4cdb-45a5-a934-83afee2ea73e" containerID="6a63199083c54b220a32e1a286bf8c279734a7bfe57fdf69418591fb2d942e14" exitCode=0 Nov 03 16:49:58.274728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:58.274633 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" event={"ID":"c0e0f140-4cdb-45a5-a934-83afee2ea73e","Type":"ContainerDied","Data":"6a63199083c54b220a32e1a286bf8c279734a7bfe57fdf69418591fb2d942e14"} Nov 03 16:49:58.276262 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:58.276243 5946 generic.go:356] "Generic (PLEG): container finished" podID="bf063e91-8e8a-4ac6-a71c-52a8c333d953" containerID="e5b5fb66c0382c00b3d0c7d15158ea5e7df485b0902b4343a8a3968296d83fb1" exitCode=0 Nov 03 16:49:58.276341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:58.276314 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" event={"ID":"bf063e91-8e8a-4ac6-a71c-52a8c333d953","Type":"ContainerDied","Data":"e5b5fb66c0382c00b3d0c7d15158ea5e7df485b0902b4343a8a3968296d83fb1"} Nov 03 16:49:59.459162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.459134 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:49:59.519307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.519285 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:49:59.592627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.592536 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-xks-ca\") pod \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " Nov 03 16:49:59.592627 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.592593 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf063e91-8e8a-4ac6-a71c-52a8c333d953-kube-api-access-nqhpg\" (UniqueName: \"kubernetes.io/projected/bf063e91-8e8a-4ac6-a71c-52a8c333d953-kube-api-access-nqhpg\") pod \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " Nov 03 16:49:59.592879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.592708 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-openshift-ca\") pod \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\" (UID: \"bf063e91-8e8a-4ac6-a71c-52a8c333d953\") " Nov 03 16:49:59.593074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.593041 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-openshift-ca" pod "bf063e91-8e8a-4ac6-a71c-52a8c333d953" (UID: "bf063e91-8e8a-4ac6-a71c-52a8c333d953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:59.593317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.593291 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.595026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.594994 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-xks-ca" pod "bf063e91-8e8a-4ac6-a71c-52a8c333d953" (UID: "bf063e91-8e8a-4ac6-a71c-52a8c333d953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:59.595026 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.595002 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf063e91-8e8a-4ac6-a71c-52a8c333d953-kube-api-access-nqhpg" pod "bf063e91-8e8a-4ac6-a71c-52a8c333d953" (UID: "bf063e91-8e8a-4ac6-a71c-52a8c333d953"). InnerVolumeSpecName "kube-api-access-nqhpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:59.619632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.619602 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:49:59.693812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.693777 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-xks-ca\") pod \"78eae67e-f178-496b-9a7d-f70f553dd5b4\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " Nov 03 16:49:59.694232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.694018 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-openshift-ca\") pod \"78eae67e-f178-496b-9a7d-f70f553dd5b4\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " Nov 03 16:49:59.694232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.694057 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78eae67e-f178-496b-9a7d-f70f553dd5b4-kube-api-access-jwfhc\" (UniqueName: \"kubernetes.io/projected/78eae67e-f178-496b-9a7d-f70f553dd5b4-kube-api-access-jwfhc\") pod \"78eae67e-f178-496b-9a7d-f70f553dd5b4\" (UID: \"78eae67e-f178-496b-9a7d-f70f553dd5b4\") " Nov 03 16:49:59.694504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.694484 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf063e91-8e8a-4ac6-a71c-52a8c333d953-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.694580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.694511 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqhpg\" (UniqueName: \"kubernetes.io/projected/bf063e91-8e8a-4ac6-a71c-52a8c333d953-kube-api-access-nqhpg\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.695151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.695104 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-openshift-ca" pod "78eae67e-f178-496b-9a7d-f70f553dd5b4" (UID: "78eae67e-f178-496b-9a7d-f70f553dd5b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:59.697870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.697845 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-xks-ca" pod "78eae67e-f178-496b-9a7d-f70f553dd5b4" (UID: "78eae67e-f178-496b-9a7d-f70f553dd5b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:59.697978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.697894 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78eae67e-f178-496b-9a7d-f70f553dd5b4-kube-api-access-jwfhc" pod "78eae67e-f178-496b-9a7d-f70f553dd5b4" (UID: "78eae67e-f178-496b-9a7d-f70f553dd5b4"). InnerVolumeSpecName "kube-api-access-jwfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:59.765662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.765634 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:49:59.795662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.795628 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-xks-ca\") pod \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " Nov 03 16:49:59.795831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.795818 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-openshift-ca\") pod \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " Nov 03 16:49:59.795880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.795870 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e0f140-4cdb-45a5-a934-83afee2ea73e-kube-api-access-7bwpg\" (UniqueName: \"kubernetes.io/projected/c0e0f140-4cdb-45a5-a934-83afee2ea73e-kube-api-access-7bwpg\") pod \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\" (UID: \"c0e0f140-4cdb-45a5-a934-83afee2ea73e\") " Nov 03 16:49:59.796171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.796157 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.796171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.796157 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-openshift-ca" pod "c0e0f140-4cdb-45a5-a934-83afee2ea73e" (UID: "c0e0f140-4cdb-45a5-a934-83afee2ea73e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:59.796171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.796171 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfhc\" (UniqueName: \"kubernetes.io/projected/78eae67e-f178-496b-9a7d-f70f553dd5b4-kube-api-access-jwfhc\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.796412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.796183 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78eae67e-f178-496b-9a7d-f70f553dd5b4-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.797847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.797825 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-xks-ca" pod "c0e0f140-4cdb-45a5-a934-83afee2ea73e" (UID: "c0e0f140-4cdb-45a5-a934-83afee2ea73e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:59.798059 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.798035 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e0f140-4cdb-45a5-a934-83afee2ea73e-kube-api-access-7bwpg" pod "c0e0f140-4cdb-45a5-a934-83afee2ea73e" (UID: "c0e0f140-4cdb-45a5-a934-83afee2ea73e"). InnerVolumeSpecName "kube-api-access-7bwpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:59.896845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.896755 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-socket\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.896845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.896811 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-data\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.896870 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-token\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-token\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.896907 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e436b7-0ecc-45f2-9418-d110fadc4e67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e436b7-0ecc-45f2-9418-d110fadc4e67-istiod-ca-cert\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.896965 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-certs\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897018 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-envoy\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897048 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-socket" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:59.897086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897070 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-podinfo\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897070 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-data" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:59.897474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897190 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-credential-socket\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897239 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-kube-api-access-qvhb4\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-kube-api-access-qvhb4\") pod \"10e436b7-0ecc-45f2-9418-d110fadc4e67\" (UID: \"10e436b7-0ecc-45f2-9418-d110fadc4e67\") " Nov 03 16:49:59.897474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897341 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-certs" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:59.897474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897451 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e436b7-0ecc-45f2-9418-d110fadc4e67-istiod-ca-cert" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:59.897679 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897500 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-credential-socket" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:59.897801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897784 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.897877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897809 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.897877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897823 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e436b7-0ecc-45f2-9418-d110fadc4e67-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.897877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897839 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bwpg\" (UniqueName: \"kubernetes.io/projected/c0e0f140-4cdb-45a5-a934-83afee2ea73e-kube-api-access-7bwpg\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.897877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897853 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.897877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897866 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0f140-4cdb-45a5-a934-83afee2ea73e-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.898080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897879 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.898080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.897894 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.899446 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.899424 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-token" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:59.899539 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.899454 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-podinfo" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:59.899738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.899723 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-envoy" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:59.899846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.899832 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-kube-api-access-qvhb4" pod "10e436b7-0ecc-45f2-9418-d110fadc4e67" (UID: "10e436b7-0ecc-45f2-9418-d110fadc4e67"). InnerVolumeSpecName "kube-api-access-qvhb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:59.969319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.969283 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9"] Nov 03 16:49:59.998801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.998773 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.998801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.998800 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhb4\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-kube-api-access-qvhb4\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.999005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.998811 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:49:59.999005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:49:59.998821 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e436b7-0ecc-45f2-9418-d110fadc4e67-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:00.166710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.166629 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:00.321693 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.321659 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" event={"ID":"78eae67e-f178-496b-9a7d-f70f553dd5b4","Type":"ContainerDied","Data":"8f230e88458127ec26d49969670f289d905a2508d1e9ee35eefecd6c3b84c940"} Nov 03 16:50:00.321898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.321879 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f230e88458127ec26d49969670f289d905a2508d1e9ee35eefecd6c3b84c940" Nov 03 16:50:00.322011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.321957 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh" Nov 03 16:50:00.324802 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.324775 5946 generic.go:356] "Generic (PLEG): container finished" podID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerID="61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3" exitCode=0 Nov 03 16:50:00.325071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.325049 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" event={"ID":"10e436b7-0ecc-45f2-9418-d110fadc4e67","Type":"ContainerDied","Data":"61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3"} Nov 03 16:50:00.325307 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.325286 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" event={"ID":"10e436b7-0ecc-45f2-9418-d110fadc4e67","Type":"ContainerDied","Data":"34033e1fadbe89576c07d87d5580e214351d3e755a89e30be7b8389778b36662"} Nov 03 16:50:00.325483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.325467 5946 scope.go:122] "RemoveContainer" containerID="61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3" Nov 03 16:50:00.325834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.325813 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6" Nov 03 16:50:00.330067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.330046 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" event={"ID":"c0e0f140-4cdb-45a5-a934-83afee2ea73e","Type":"ContainerDied","Data":"72092c22dfe7adde5d3ad95ccb9ae80190b6665c128f577ebb0cbc2dacc6fc2c"} Nov 03 16:50:00.330318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.330196 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72092c22dfe7adde5d3ad95ccb9ae80190b6665c128f577ebb0cbc2dacc6fc2c" Nov 03 16:50:00.330561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.330544 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4" Nov 03 16:50:00.334564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.334549 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" Nov 03 16:50:00.334728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.334702 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr" event={"ID":"bf063e91-8e8a-4ac6-a71c-52a8c333d953","Type":"ContainerDied","Data":"2ebb845f14599be30d690737764221319cf41d63609544c924aa9ccc317e5dba"} Nov 03 16:50:00.334801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.334756 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ebb845f14599be30d690737764221319cf41d63609544c924aa9ccc317e5dba" Nov 03 16:50:00.346186 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.346166 5946 scope.go:122] "RemoveContainer" containerID="df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70" Nov 03 16:50:00.357605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.357581 5946 scope.go:122] "RemoveContainer" containerID="61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3" Nov 03 16:50:00.357951 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:00.357929 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3\": container with ID starting with 61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3 not found: ID does not exist" containerID="61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3" Nov 03 16:50:00.358060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.357962 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3"} err="failed to get container status \"61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3\": rpc error: code = NotFound desc = could not find container \"61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3\": container with ID starting with 61956828b8f4eda664667b573a03f56d9d3fe86dbd959b70a36d512ab31254f3 not found: ID does not exist" Nov 03 16:50:00.358060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.357987 5946 scope.go:122] "RemoveContainer" containerID="df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70" Nov 03 16:50:00.358674 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:00.358652 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70\": container with ID starting with df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70 not found: ID does not exist" containerID="df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70" Nov 03 16:50:00.358788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.358679 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70"} err="failed to get container status \"df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70\": rpc error: code = NotFound desc = could not find container \"df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70\": container with ID starting with df10fc5fa85106aa2559f9de38bd9a5fec85b8c0d5d680b49a245470b190cd70 not found: ID does not exist" Nov 03 16:50:00.603733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.603698 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6"] Nov 03 16:50:00.612909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.612883 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0ea8a5-openshift-default-5cbb47bbb9-lw6d6"] Nov 03 16:50:00.703033 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.702941 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:01.012287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:00.993665 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc"] Nov 03 16:50:01.067046 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:01.066437 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" containerID="cri-o://3c229fbe88978a18ebc66c5e30bf454e582b731b65c3308d968e57f3a2aaf482" gracePeriod=30 Nov 03 16:50:02.112024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.111982 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" podUID="5078901f-7793-4090-8611-36726bb133c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:50:02.112761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.112323 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" podUID="aad44a8d-f155-4329-afd7-ceaed3489ede" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:50:02.117578 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.117546 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" path="/var/lib/kubelet/pods/10e436b7-0ecc-45f2-9418-d110fadc4e67/volumes" Nov 03 16:50:02.260600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.260444 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:02.278200 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278167 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp"] Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278699 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c0e0f140-4cdb-45a5-a934-83afee2ea73e" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278713 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e0f140-4cdb-45a5-a934-83afee2ea73e" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278724 5946 cpu_manager.go:414] "containerMap: removing container" podUID="78eae67e-f178-496b-9a7d-f70f553dd5b4" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278730 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78eae67e-f178-496b-9a7d-f70f553dd5b4" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278743 5946 cpu_manager.go:414] "containerMap: removing container" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278748 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278767 5946 cpu_manager.go:414] "containerMap: removing container" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278771 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278790 5946 cpu_manager.go:414] "containerMap: removing container" podUID="bf063e91-8e8a-4ac6-a71c-52a8c333d953" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278796 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf063e91-8e8a-4ac6-a71c-52a8c333d953" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278874 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf063e91-8e8a-4ac6-a71c-52a8c333d953" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278887 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e0f140-4cdb-45a5-a934-83afee2ea73e" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278892 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="78eae67e-f178-496b-9a7d-f70f553dd5b4" containerName="revoke-keys" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278898 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" Nov 03 16:50:02.279190 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.278906 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e436b7-0ecc-45f2-9418-d110fadc4e67" containerName="istio-proxy" Nov 03 16:50:02.283873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.283849 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.346796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.346768 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:50:02.347071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.346907 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:50:02.347071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.346976 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:50:02.347071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.347045 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:50:02.349107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.349078 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp"] Nov 03 16:50:02.359974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.359930 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vq2d\" (UniqueName: \"kubernetes.io/projected/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-kube-api-access-6vq2d\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.360138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.360112 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.360283 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.360266 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.461704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.461618 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.461865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.461718 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.461865 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.461796 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vq2d\" (UniqueName: \"kubernetes.io/projected/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-kube-api-access-6vq2d\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.462402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.462376 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.464318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.464288 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.522610 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.522553 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vq2d\" (UniqueName: \"kubernetes.io/projected/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-kube-api-access-6vq2d\") pod \"maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.599384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.599343 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:02.998970 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:02.998937 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp"] Nov 03 16:50:02.999631 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:02.999602 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdbb5ff_b2fc_448c_9766_3086cc9ba55c.slice/crio-5aaa3e9abf161ec637bea1378e6fed1f7aea0c1bf95b79af1e8d23dbd6c4cefc WatchSource:0}: Error finding container 5aaa3e9abf161ec637bea1378e6fed1f7aea0c1bf95b79af1e8d23dbd6c4cefc: Status 404 returned error can't find the container with id 5aaa3e9abf161ec637bea1378e6fed1f7aea0c1bf95b79af1e8d23dbd6c4cefc Nov 03 16:50:03.418480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:03.418444 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" event={"ID":"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c","Type":"ContainerStarted","Data":"5aaa3e9abf161ec637bea1378e6fed1f7aea0c1bf95b79af1e8d23dbd6c4cefc"} Nov 03 16:50:04.427560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:04.427481 5946 generic.go:356] "Generic (PLEG): container finished" podID="3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" containerID="1a8f4e494dae4ee300199da31a81ed689a2b17f98d3bf7e65358322fc99ec73c" exitCode=0 Nov 03 16:50:04.427560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:04.427530 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" event={"ID":"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c","Type":"ContainerDied","Data":"1a8f4e494dae4ee300199da31a81ed689a2b17f98d3bf7e65358322fc99ec73c"} Nov 03 16:50:05.590220 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.590183 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:05.706973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.706880 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-kube-api-access-6vq2d\" (UniqueName: \"kubernetes.io/projected/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-kube-api-access-6vq2d\") pod \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " Nov 03 16:50:05.707228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.706974 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-openshift-ca\") pod \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " Nov 03 16:50:05.707228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.707155 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-xks-ca\") pod \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\" (UID: \"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c\") " Nov 03 16:50:05.707380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.707345 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-openshift-ca" pod "3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" (UID: "3fdbb5ff-b2fc-448c-9766-3086cc9ba55c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:05.707581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.707559 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:05.709266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.709242 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-xks-ca" pod "3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" (UID: "3fdbb5ff-b2fc-448c-9766-3086cc9ba55c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:05.709381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.709361 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-kube-api-access-6vq2d" pod "3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" (UID: "3fdbb5ff-b2fc-448c-9766-3086cc9ba55c"). InnerVolumeSpecName "kube-api-access-6vq2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:05.809899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.809869 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:05.809899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:05.809899 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vq2d\" (UniqueName: \"kubernetes.io/projected/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c-kube-api-access-6vq2d\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:06.443373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.443293 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" Nov 03 16:50:06.443373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.443333 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp" event={"ID":"3fdbb5ff-b2fc-448c-9766-3086cc9ba55c","Type":"ContainerDied","Data":"5aaa3e9abf161ec637bea1378e6fed1f7aea0c1bf95b79af1e8d23dbd6c4cefc"} Nov 03 16:50:06.443566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.443412 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aaa3e9abf161ec637bea1378e6fed1f7aea0c1bf95b79af1e8d23dbd6c4cefc" Nov 03 16:50:06.445525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.445488 5946 generic.go:356] "Generic (PLEG): container finished" podID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerID="3c229fbe88978a18ebc66c5e30bf454e582b731b65c3308d968e57f3a2aaf482" exitCode=0 Nov 03 16:50:06.445680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.445592 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" event={"ID":"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a","Type":"ContainerDied","Data":"3c229fbe88978a18ebc66c5e30bf454e582b731b65c3308d968e57f3a2aaf482"} Nov 03 16:50:06.445680 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.445671 5946 scope.go:122] "RemoveContainer" containerID="6354178bb1b7a787fc607209340afaa73ab32dc200e05d774d32ec5d30fe628f" Nov 03 16:50:06.502803 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.502767 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv"] Nov 03 16:50:06.553625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.553594 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9"] Nov 03 16:50:06.743908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.743880 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:06.863812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:06.863790 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:50:07.025848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.025804 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-certs\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.025917 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istiod-ca-cert\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.025976 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-data\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026057 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-token\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026168 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-kube-api-access-qtgvg\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-kube-api-access-qtgvg\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.025975 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-certs" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:07.026302 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026158 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-data" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:07.026469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026305 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-credential-socket\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026349 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-socket\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026388 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-podinfo\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026393 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istiod-ca-cert" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:07.026469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026456 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-envoy\") pod \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\" (UID: \"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a\") " Nov 03 16:50:07.026748 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026693 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-socket" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:07.026811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.026762 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-credential-socket" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:07.027554 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.027526 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.027671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.027556 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.027671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.027573 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.027671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.027588 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.027671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.027601 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.029303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.029269 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-kube-api-access-qtgvg" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "kube-api-access-qtgvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:07.029303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.029271 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-podinfo" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:50:07.029483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.029375 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-token" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:07.029649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.029625 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-envoy" pod "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" (UID: "3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:07.128461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.128428 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.128461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.128459 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtgvg\" (UniqueName: \"kubernetes.io/projected/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-kube-api-access-qtgvg\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.128461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.128470 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.128461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.128479 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.337540 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.337489 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:07.457001 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.456963 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" event={"ID":"3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a","Type":"ContainerDied","Data":"a3639497b9a09231dc32cd8202ca7dc744271c5bd5d8f84bc054b3bacaea6657"} Nov 03 16:50:07.457161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.457032 5946 scope.go:122] "RemoveContainer" containerID="3c229fbe88978a18ebc66c5e30bf454e582b731b65c3308d968e57f3a2aaf482" Nov 03 16:50:07.457161 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.456975 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc" Nov 03 16:50:07.459483 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.459453 5946 generic.go:356] "Generic (PLEG): container finished" podID="5078901f-7793-4090-8611-36726bb133c7" containerID="a9053269f29c18093e25910de20daf304aaed92e383433297d9aad9a22aa1c5d" exitCode=0 Nov 03 16:50:07.459653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.459637 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" event={"ID":"5078901f-7793-4090-8611-36726bb133c7","Type":"ContainerDied","Data":"a9053269f29c18093e25910de20daf304aaed92e383433297d9aad9a22aa1c5d"} Nov 03 16:50:07.591847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.591814 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:50:07.633568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.633534 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc"] Nov 03 16:50:07.683797 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.683765 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0fda73-openshift-default-7c56c64db5-tckbc"] Nov 03 16:50:07.694922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.694896 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:50:07.738007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.737978 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5078901f-7793-4090-8611-36726bb133c7-kube-api-access-vm6cx\" (UniqueName: \"kubernetes.io/projected/5078901f-7793-4090-8611-36726bb133c7-kube-api-access-vm6cx\") pod \"5078901f-7793-4090-8611-36726bb133c7\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " Nov 03 16:50:07.738198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.738182 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5078901f-7793-4090-8611-36726bb133c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5078901f-7793-4090-8611-36726bb133c7-plugins-config-volume\") pod \"5078901f-7793-4090-8611-36726bb133c7\" (UID: \"5078901f-7793-4090-8611-36726bb133c7\") " Nov 03 16:50:07.738599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.738576 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5078901f-7793-4090-8611-36726bb133c7-plugins-config-volume" pod "5078901f-7793-4090-8611-36726bb133c7" (UID: "5078901f-7793-4090-8611-36726bb133c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:07.738725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.738709 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5078901f-7793-4090-8611-36726bb133c7-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.740280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.740259 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5078901f-7793-4090-8611-36726bb133c7-kube-api-access-vm6cx" pod "5078901f-7793-4090-8611-36726bb133c7" (UID: "5078901f-7793-4090-8611-36726bb133c7"). InnerVolumeSpecName "kube-api-access-vm6cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:07.840068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.839985 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad44a8d-f155-4329-afd7-ceaed3489ede-kube-api-access-cwrmj\" (UniqueName: \"kubernetes.io/projected/aad44a8d-f155-4329-afd7-ceaed3489ede-kube-api-access-cwrmj\") pod \"aad44a8d-f155-4329-afd7-ceaed3489ede\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " Nov 03 16:50:07.840567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.840076 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad44a8d-f155-4329-afd7-ceaed3489ede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad44a8d-f155-4329-afd7-ceaed3489ede-plugins-config-volume\") pod \"aad44a8d-f155-4329-afd7-ceaed3489ede\" (UID: \"aad44a8d-f155-4329-afd7-ceaed3489ede\") " Nov 03 16:50:07.840646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.840567 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad44a8d-f155-4329-afd7-ceaed3489ede-plugins-config-volume" pod "aad44a8d-f155-4329-afd7-ceaed3489ede" (UID: "aad44a8d-f155-4329-afd7-ceaed3489ede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:07.840646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.840631 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6cx\" (UniqueName: \"kubernetes.io/projected/5078901f-7793-4090-8611-36726bb133c7-kube-api-access-vm6cx\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.842418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.842392 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad44a8d-f155-4329-afd7-ceaed3489ede-kube-api-access-cwrmj" pod "aad44a8d-f155-4329-afd7-ceaed3489ede" (UID: "aad44a8d-f155-4329-afd7-ceaed3489ede"). InnerVolumeSpecName "kube-api-access-cwrmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:07.942009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.941972 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrmj\" (UniqueName: \"kubernetes.io/projected/aad44a8d-f155-4329-afd7-ceaed3489ede-kube-api-access-cwrmj\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:07.942009 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:07.942007 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad44a8d-f155-4329-afd7-ceaed3489ede-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:08.005412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.005380 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" path="/var/lib/kubelet/pods/3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a/volumes" Nov 03 16:50:08.470429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.470394 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" Nov 03 16:50:08.470429 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.470413 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4" event={"ID":"5078901f-7793-4090-8611-36726bb133c7","Type":"ContainerDied","Data":"5e791006a5c564784f7a53f714f0c6ddd8dd45cec4251b7f9187e07c6b1a72fb"} Nov 03 16:50:08.470669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.470475 5946 scope.go:122] "RemoveContainer" containerID="a9053269f29c18093e25910de20daf304aaed92e383433297d9aad9a22aa1c5d" Nov 03 16:50:08.471984 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.471962 5946 generic.go:356] "Generic (PLEG): container finished" podID="aad44a8d-f155-4329-afd7-ceaed3489ede" containerID="71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e" exitCode=0 Nov 03 16:50:08.472144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.472130 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" event={"ID":"aad44a8d-f155-4329-afd7-ceaed3489ede","Type":"ContainerDied","Data":"71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e"} Nov 03 16:50:08.472187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.472169 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" event={"ID":"aad44a8d-f155-4329-afd7-ceaed3489ede","Type":"ContainerDied","Data":"2e9a3376eefbfc7caeb1e7c7949e138eec1c5d58a91141dce66590df11ba42d4"} Nov 03 16:50:08.472278 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.472266 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj" Nov 03 16:50:08.485293 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.485268 5946 scope.go:122] "RemoveContainer" containerID="71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e" Nov 03 16:50:08.495583 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.495557 5946 scope.go:122] "RemoveContainer" containerID="71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e" Nov 03 16:50:08.495877 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:08.495856 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e\": container with ID starting with 71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e not found: ID does not exist" containerID="71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e" Nov 03 16:50:08.495956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.495885 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e"} err="failed to get container status \"71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e\": rpc error: code = NotFound desc = could not find container \"71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e\": container with ID starting with 71bda95cbde31b5d5873e1e8a3a31a212d70f5b7d24b867086a8818f3dbc700e not found: ID does not exist" Nov 03 16:50:08.643239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.643189 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4"] Nov 03 16:50:08.647721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.647696 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-547a38-7dd9b655b7-nlqx4"] Nov 03 16:50:08.767784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.767753 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj"] Nov 03 16:50:08.794507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:08.794476 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-547a38-59c6d9987f-dmrwj"] Nov 03 16:50:10.005689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:10.005655 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5078901f-7793-4090-8611-36726bb133c7" path="/var/lib/kubelet/pods/5078901f-7793-4090-8611-36726bb133c7/volumes" Nov 03 16:50:10.006391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:10.006372 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad44a8d-f155-4329-afd7-ceaed3489ede" path="/var/lib/kubelet/pods/aad44a8d-f155-4329-afd7-ceaed3489ede/volumes" Nov 03 16:50:10.047431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:10.047398 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:10.246667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:10.246632 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:11.740011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:11.739895 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:11.920491 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:11.920462 5946 scope.go:122] "RemoveContainer" containerID="84dd1ac35b0c3dd40e5ed9e0c4e3a16a580e3ceea5c0847054a4e30a6ea55de8" Nov 03 16:50:11.931709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:11.931667 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:11.933334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:11.933290 5946 scope.go:122] "RemoveContainer" containerID="a20a0eefdb361f8bf148bb160693374fcc5000d971df5e7b12c229e16744391b" Nov 03 16:50:11.943656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:11.943634 5946 scope.go:122] "RemoveContainer" containerID="54204414457e1532274c571ab1963ee405971e34d712eb7503f7481c207fd656" Nov 03 16:50:11.977124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:11.975779 5946 scope.go:122] "RemoveContainer" containerID="5fab07e37ab4cdee8d153182d11c49f4470547a6cef083e7703e7484e84d7da2" Nov 03 16:50:12.587308 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:12.587265 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:50:13.505757 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:13.505720 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:13.715678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:13.715638 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:13.932699 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:13.932605 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:13.966344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:13.966308 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c"] Nov 03 16:50:14.017715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:14.017679 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c626cd-qgq2c"] Nov 03 16:50:14.333814 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:14.333787 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:14.536020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:14.535982 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:14.992674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:14.992626 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" containerID="cri-o://aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d" gracePeriod=45 Nov 03 16:50:15.271319 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.271295 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:50:15.352987 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.352952 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-tmp-dir\") pod \"e310e79e-049f-407b-82f7-bcd67c882fa8\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " Nov 03 16:50:15.353180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353103 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-dshm\") pod \"e310e79e-049f-407b-82f7-bcd67c882fa8\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " Nov 03 16:50:15.353180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353157 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e310e79e-049f-407b-82f7-bcd67c882fa8-tls-certs\" (UniqueName: \"kubernetes.io/secret/e310e79e-049f-407b-82f7-bcd67c882fa8-tls-certs\") pod \"e310e79e-049f-407b-82f7-bcd67c882fa8\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " Nov 03 16:50:15.353325 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353262 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-model-cache\") pod \"e310e79e-049f-407b-82f7-bcd67c882fa8\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " Nov 03 16:50:15.353325 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353282 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-tmp-dir" pod "e310e79e-049f-407b-82f7-bcd67c882fa8" (UID: "e310e79e-049f-407b-82f7-bcd67c882fa8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:15.353441 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353325 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-home\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-home\") pod \"e310e79e-049f-407b-82f7-bcd67c882fa8\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " Nov 03 16:50:15.353441 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353368 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e310e79e-049f-407b-82f7-bcd67c882fa8-kube-api-access-79rlb\" (UniqueName: \"kubernetes.io/projected/e310e79e-049f-407b-82f7-bcd67c882fa8-kube-api-access-79rlb\") pod \"e310e79e-049f-407b-82f7-bcd67c882fa8\" (UID: \"e310e79e-049f-407b-82f7-bcd67c882fa8\") " Nov 03 16:50:15.353579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353548 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-model-cache" pod "e310e79e-049f-407b-82f7-bcd67c882fa8" (UID: "e310e79e-049f-407b-82f7-bcd67c882fa8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:15.353631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353578 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-home" pod "e310e79e-049f-407b-82f7-bcd67c882fa8" (UID: "e310e79e-049f-407b-82f7-bcd67c882fa8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:15.353827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353804 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:15.353827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353825 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:15.353972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.353833 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:15.355659 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.355628 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e310e79e-049f-407b-82f7-bcd67c882fa8-tls-certs" pod "e310e79e-049f-407b-82f7-bcd67c882fa8" (UID: "e310e79e-049f-407b-82f7-bcd67c882fa8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:15.355743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.355669 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-dshm" pod "e310e79e-049f-407b-82f7-bcd67c882fa8" (UID: "e310e79e-049f-407b-82f7-bcd67c882fa8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:15.355887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.355867 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e310e79e-049f-407b-82f7-bcd67c882fa8-kube-api-access-79rlb" pod "e310e79e-049f-407b-82f7-bcd67c882fa8" (UID: "e310e79e-049f-407b-82f7-bcd67c882fa8"). InnerVolumeSpecName "kube-api-access-79rlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:15.454940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.454911 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rlb\" (UniqueName: \"kubernetes.io/projected/e310e79e-049f-407b-82f7-bcd67c882fa8-kube-api-access-79rlb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:15.454940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.454938 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e310e79e-049f-407b-82f7-bcd67c882fa8-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:15.454940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.454949 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e310e79e-049f-407b-82f7-bcd67c882fa8-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:15.591169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.591064 5946 generic.go:356] "Generic (PLEG): container finished" podID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerID="aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d" exitCode=0 Nov 03 16:50:15.593925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.591842 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" event={"ID":"e310e79e-049f-407b-82f7-bcd67c882fa8","Type":"ContainerDied","Data":"aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d"} Nov 03 16:50:15.593925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.591891 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" event={"ID":"e310e79e-049f-407b-82f7-bcd67c882fa8","Type":"ContainerDied","Data":"c7926b7e1d8e67ad811bd168fbd8ecb6f720799c27a60b8a42d3419ba0f25c72"} Nov 03 16:50:15.593925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.591914 5946 scope.go:122] "RemoveContainer" containerID="aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d" Nov 03 16:50:15.593925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.592067 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9" Nov 03 16:50:15.631632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.631588 5946 scope.go:122] "RemoveContainer" containerID="aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d" Nov 03 16:50:15.632147 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:15.632121 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d\": container with ID starting with aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d not found: ID does not exist" containerID="aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d" Nov 03 16:50:15.632295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.632175 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d"} err="failed to get container status \"aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d\": rpc error: code = NotFound desc = could not find container \"aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d\": container with ID starting with aebf561c8401417eaa9bef7897d72a146de6dfcb6a24564792db716d67877c6d not found: ID does not exist" Nov 03 16:50:15.785094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.785055 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9"] Nov 03 16:50:15.834891 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:15.834855 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-192564/simulator-w3-192564-kserve-567d666b65-qmgk9"] Nov 03 16:50:16.005641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:16.005606 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6" path="/var/lib/kubelet/pods/04de1a84-b1e2-4fe1-b1fa-186f4a60e4b6/volumes" Nov 03 16:50:16.006151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:16.006133 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" path="/var/lib/kubelet/pods/e310e79e-049f-407b-82f7-bcd67c882fa8/volumes" Nov 03 16:50:16.339792 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:16.339715 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:18.336522 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.336471 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2"] Nov 03 16:50:18.337019 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337004 5946 cpu_manager.go:414] "containerMap: removing container" podUID="aad44a8d-f155-4329-afd7-ceaed3489ede" containerName="payload-pre-processing" Nov 03 16:50:18.337071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337020 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad44a8d-f155-4329-afd7-ceaed3489ede" containerName="payload-pre-processing" Nov 03 16:50:18.337071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337043 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" containerName="revoke-keys" Nov 03 16:50:18.337071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337049 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" containerName="revoke-keys" Nov 03 16:50:18.337071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337061 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" Nov 03 16:50:18.337071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337066 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" Nov 03 16:50:18.337240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337080 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" Nov 03 16:50:18.337240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337085 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" Nov 03 16:50:18.337240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337098 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" Nov 03 16:50:18.337240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337106 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" Nov 03 16:50:18.337240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337126 5946 cpu_manager.go:414] "containerMap: removing container" podUID="5078901f-7793-4090-8611-36726bb133c7" containerName="payload-processing" Nov 03 16:50:18.337240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337130 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5078901f-7793-4090-8611-36726bb133c7" containerName="payload-processing" Nov 03 16:50:18.337240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337234 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="5078901f-7793-4090-8611-36726bb133c7" containerName="payload-processing" Nov 03 16:50:18.337459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337247 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad44a8d-f155-4329-afd7-ceaed3489ede" containerName="payload-pre-processing" Nov 03 16:50:18.337459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337254 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" containerName="revoke-keys" Nov 03 16:50:18.337459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337262 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" Nov 03 16:50:18.337459 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.337273 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e310e79e-049f-407b-82f7-bcd67c882fa8" containerName="main" Nov 03 16:50:18.342187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.342169 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.393509 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.393465 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2"] Nov 03 16:50:18.394333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.394301 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd0e137-21ed-4af7-a830-fcb0b0ac2e3a" containerName="istio-proxy" Nov 03 16:50:18.396978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.396941 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsgh7\" (UniqueName: \"kubernetes.io/projected/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-kube-api-access-hsgh7\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.396978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.396975 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.397165 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.397097 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.397683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.397664 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.400949 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.400924 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2"] Nov 03 16:50:18.441602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.441570 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:50:18.441602 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.441575 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:50:18.448798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.448765 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:50:18.448798 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.448775 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:50:18.475979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.475938 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2"] Nov 03 16:50:18.498265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.498226 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjhc\" (UniqueName: \"kubernetes.io/projected/8547b3c3-1562-49f9-ab86-cf7a513f9348-kube-api-access-fhjhc\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.498437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.498300 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgh7\" (UniqueName: \"kubernetes.io/projected/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-kube-api-access-hsgh7\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.498437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.498366 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.498437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.498427 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.498635 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.498499 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.498690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.498635 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.499732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.499702 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.502285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.502257 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.560412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.560379 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgh7\" (UniqueName: \"kubernetes.io/projected/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-kube-api-access-hsgh7\") pod \"maas-api-revoke-sub-b964ed13-5b6t2\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.593274 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.593182 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:18.600020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.599992 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.600179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.600095 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjhc\" (UniqueName: \"kubernetes.io/projected/8547b3c3-1562-49f9-ab86-cf7a513f9348-kube-api-access-fhjhc\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.600354 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.600252 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.600690 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.600672 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.602871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.602845 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.656454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.656419 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:18.667496 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.667462 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjhc\" (UniqueName: \"kubernetes.io/projected/8547b3c3-1562-49f9-ab86-cf7a513f9348-kube-api-access-fhjhc\") pod \"maas-api-revoke-sub-69da120b-ssqx2\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:18.713367 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:18.713339 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:19.089566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.089539 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2"] Nov 03 16:50:19.090710 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:19.090685 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f00d68_0ad4_4d1c_83b8_ce2500abb5c4.slice/crio-f18f42618a13abd634b1cbcb9bc447f8ecabbae5e3ccc695a6e56023ddb1a2c2 WatchSource:0}: Error finding container f18f42618a13abd634b1cbcb9bc447f8ecabbae5e3ccc695a6e56023ddb1a2c2: Status 404 returned error can't find the container with id f18f42618a13abd634b1cbcb9bc447f8ecabbae5e3ccc695a6e56023ddb1a2c2 Nov 03 16:50:19.138174 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.138149 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2"] Nov 03 16:50:19.139364 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:19.139341 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8547b3c3_1562_49f9_ab86_cf7a513f9348.slice/crio-458a139a134c8090528415f081967b8ecbea73b23a9e5f7aab37d54614b48631 WatchSource:0}: Error finding container 458a139a134c8090528415f081967b8ecbea73b23a9e5f7aab37d54614b48631: Status 404 returned error can't find the container with id 458a139a134c8090528415f081967b8ecbea73b23a9e5f7aab37d54614b48631 Nov 03 16:50:19.632050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.632006 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" event={"ID":"8547b3c3-1562-49f9-ab86-cf7a513f9348","Type":"ContainerStarted","Data":"969cce4f0227841f4966c64e8b0eb09cb9fa3b5cd83921514d5942c6ca64500b"} Nov 03 16:50:19.632529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.632076 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" event={"ID":"8547b3c3-1562-49f9-ab86-cf7a513f9348","Type":"ContainerStarted","Data":"458a139a134c8090528415f081967b8ecbea73b23a9e5f7aab37d54614b48631"} Nov 03 16:50:19.633815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.633780 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" event={"ID":"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4","Type":"ContainerStarted","Data":"e2ad5c1936aa24a448e23d09c535d4a1487460f82eaa4d7662e2272c997a0681"} Nov 03 16:50:19.633951 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.633836 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" event={"ID":"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4","Type":"ContainerStarted","Data":"f18f42618a13abd634b1cbcb9bc447f8ecabbae5e3ccc695a6e56023ddb1a2c2"} Nov 03 16:50:19.743670 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.743636 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:19.759960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.759908 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" podStartSLOduration=1.759893079 podStartE2EDuration="1.759893079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:19.756167656 +0000 UTC m=+2289.877220479" watchObservedRunningTime="2026-11-03 16:50:19.759893079 +0000 UTC m=+2289.880945901" Nov 03 16:50:19.880404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:19.880303 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" podStartSLOduration=1.880286847 podStartE2EDuration="1.880286847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:19.877894042 +0000 UTC m=+2289.998946865" watchObservedRunningTime="2026-11-03 16:50:19.880286847 +0000 UTC m=+2290.001339670" Nov 03 16:50:20.194715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.194686 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:20.405733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.405695 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:20.603641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.603610 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:20.642531 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.642499 5946 generic.go:356] "Generic (PLEG): container finished" podID="8547b3c3-1562-49f9-ab86-cf7a513f9348" containerID="969cce4f0227841f4966c64e8b0eb09cb9fa3b5cd83921514d5942c6ca64500b" exitCode=0 Nov 03 16:50:20.642937 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.642580 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" event={"ID":"8547b3c3-1562-49f9-ab86-cf7a513f9348","Type":"ContainerDied","Data":"969cce4f0227841f4966c64e8b0eb09cb9fa3b5cd83921514d5942c6ca64500b"} Nov 03 16:50:20.644192 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.644164 5946 generic.go:356] "Generic (PLEG): container finished" podID="70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" containerID="e2ad5c1936aa24a448e23d09c535d4a1487460f82eaa4d7662e2272c997a0681" exitCode=0 Nov 03 16:50:20.644329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.644232 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" event={"ID":"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4","Type":"ContainerDied","Data":"e2ad5c1936aa24a448e23d09c535d4a1487460f82eaa4d7662e2272c997a0681"} Nov 03 16:50:20.657193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.657172 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl"] Nov 03 16:50:20.708091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:20.708065 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-66f145-ss6bl"] Nov 03 16:50:21.042894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.042862 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:21.261125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.261096 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:21.524273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.524183 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" containerID="cri-o://32d280cb2c7e356d1e9b25a1ff2678189dda4ac510d38d1d2eb6d03d4f6b2752" gracePeriod=45 Nov 03 16:50:21.574402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.574371 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" containerID="cri-o://08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01" gracePeriod=45 Nov 03 16:50:21.653685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.653635 5946 generic.go:356] "Generic (PLEG): container finished" podID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerID="32d280cb2c7e356d1e9b25a1ff2678189dda4ac510d38d1d2eb6d03d4f6b2752" exitCode=0 Nov 03 16:50:21.654092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.653700 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" event={"ID":"424b9fdd-1579-4162-827c-e48d0c5af5c9","Type":"ContainerDied","Data":"32d280cb2c7e356d1e9b25a1ff2678189dda4ac510d38d1d2eb6d03d4f6b2752"} Nov 03 16:50:21.713135 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.711763 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused" Nov 03 16:50:21.718472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.718268 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": dial tcp 10.134.0.91:8000: connect: connection refused" Nov 03 16:50:21.850727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.850696 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:21.862252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.862196 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:21.871417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.871393 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:50:21.906335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.906308 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:50:21.943758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.943719 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424b9fdd-1579-4162-827c-e48d0c5af5c9-kube-api-access-vm766\" (UniqueName: \"kubernetes.io/projected/424b9fdd-1579-4162-827c-e48d0c5af5c9-kube-api-access-vm766\") pod \"424b9fdd-1579-4162-827c-e48d0c5af5c9\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " Nov 03 16:50:21.943938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.943774 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8547b3c3-1562-49f9-ab86-cf7a513f9348-kube-api-access-fhjhc\" (UniqueName: \"kubernetes.io/projected/8547b3c3-1562-49f9-ab86-cf7a513f9348-kube-api-access-fhjhc\") pod \"8547b3c3-1562-49f9-ab86-cf7a513f9348\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " Nov 03 16:50:21.943938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.943832 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-home\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-home\") pod \"424b9fdd-1579-4162-827c-e48d0c5af5c9\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " Nov 03 16:50:21.943938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.943881 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-xks-ca\") pod \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " Nov 03 16:50:21.943938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.943908 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-xks-ca\") pod \"8547b3c3-1562-49f9-ab86-cf7a513f9348\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " Nov 03 16:50:21.944222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.943968 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-openshift-ca\") pod \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " Nov 03 16:50:21.944222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944055 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424b9fdd-1579-4162-827c-e48d0c5af5c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/424b9fdd-1579-4162-827c-e48d0c5af5c9-tls-certs\") pod \"424b9fdd-1579-4162-827c-e48d0c5af5c9\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " Nov 03 16:50:21.944222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944087 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-kube-api-access-hsgh7\" (UniqueName: \"kubernetes.io/projected/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-kube-api-access-hsgh7\") pod \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\" (UID: \"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4\") " Nov 03 16:50:21.944222 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944158 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-tmp-dir\") pod \"424b9fdd-1579-4162-827c-e48d0c5af5c9\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " Nov 03 16:50:21.944500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944241 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-home" pod "424b9fdd-1579-4162-827c-e48d0c5af5c9" (UID: "424b9fdd-1579-4162-827c-e48d0c5af5c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:21.944500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944301 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-model-cache\") pod \"424b9fdd-1579-4162-827c-e48d0c5af5c9\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " Nov 03 16:50:21.944500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944343 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-openshift-ca\") pod \"8547b3c3-1562-49f9-ab86-cf7a513f9348\" (UID: \"8547b3c3-1562-49f9-ab86-cf7a513f9348\") " Nov 03 16:50:21.944500 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944408 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-dshm\") pod \"424b9fdd-1579-4162-827c-e48d0c5af5c9\" (UID: \"424b9fdd-1579-4162-827c-e48d0c5af5c9\") " Nov 03 16:50:21.944724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.944689 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-openshift-ca" pod "70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" (UID: "70f00d68-0ad4-4d1c-83b8-ce2500abb5c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:21.945271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.945243 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:21.945400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.945275 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:21.945642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.945591 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-model-cache" pod "424b9fdd-1579-4162-827c-e48d0c5af5c9" (UID: "424b9fdd-1579-4162-827c-e48d0c5af5c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:21.945878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.945853 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-tmp-dir" pod "424b9fdd-1579-4162-827c-e48d0c5af5c9" (UID: "424b9fdd-1579-4162-827c-e48d0c5af5c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:21.946193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.946171 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-openshift-ca" pod "8547b3c3-1562-49f9-ab86-cf7a513f9348" (UID: "8547b3c3-1562-49f9-ab86-cf7a513f9348"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:21.946813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.946786 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424b9fdd-1579-4162-827c-e48d0c5af5c9-kube-api-access-vm766" pod "424b9fdd-1579-4162-827c-e48d0c5af5c9" (UID: "424b9fdd-1579-4162-827c-e48d0c5af5c9"). InnerVolumeSpecName "kube-api-access-vm766". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:21.947444 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.947423 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8547b3c3-1562-49f9-ab86-cf7a513f9348-kube-api-access-fhjhc" pod "8547b3c3-1562-49f9-ab86-cf7a513f9348" (UID: "8547b3c3-1562-49f9-ab86-cf7a513f9348"). InnerVolumeSpecName "kube-api-access-fhjhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:21.947722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.947697 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-kube-api-access-hsgh7" pod "70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" (UID: "70f00d68-0ad4-4d1c-83b8-ce2500abb5c4"). InnerVolumeSpecName "kube-api-access-hsgh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:21.947814 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.947762 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-dshm" pod "424b9fdd-1579-4162-827c-e48d0c5af5c9" (UID: "424b9fdd-1579-4162-827c-e48d0c5af5c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:21.947920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.947895 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-xks-ca" pod "8547b3c3-1562-49f9-ab86-cf7a513f9348" (UID: "8547b3c3-1562-49f9-ab86-cf7a513f9348"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:21.948108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.948089 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424b9fdd-1579-4162-827c-e48d0c5af5c9-tls-certs" pod "424b9fdd-1579-4162-827c-e48d0c5af5c9" (UID: "424b9fdd-1579-4162-827c-e48d0c5af5c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:21.948661 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:21.948644 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-xks-ca" pod "70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" (UID: "70f00d68-0ad4-4d1c-83b8-ce2500abb5c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:22.004844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.004810 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e093780f-d756-4959-b202-b772c21aed17" path="/var/lib/kubelet/pods/e093780f-d756-4959-b202-b772c21aed17/volumes" Nov 03 16:50:22.046398 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.046365 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-tmp-dir\") pod \"28625d13-a0fa-4bad-919b-3c325c396db2\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " Nov 03 16:50:22.046576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.046415 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-dshm\") pod \"28625d13-a0fa-4bad-919b-3c325c396db2\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " Nov 03 16:50:22.046576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.046455 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28625d13-a0fa-4bad-919b-3c325c396db2-tls-certs\" (UniqueName: \"kubernetes.io/secret/28625d13-a0fa-4bad-919b-3c325c396db2-tls-certs\") pod \"28625d13-a0fa-4bad-919b-3c325c396db2\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " Nov 03 16:50:22.046576 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.046536 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28625d13-a0fa-4bad-919b-3c325c396db2-kube-api-access-bhhjf\" (UniqueName: \"kubernetes.io/projected/28625d13-a0fa-4bad-919b-3c325c396db2-kube-api-access-bhhjf\") pod \"28625d13-a0fa-4bad-919b-3c325c396db2\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " Nov 03 16:50:22.046755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.046600 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-home\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-home\") pod \"28625d13-a0fa-4bad-919b-3c325c396db2\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " Nov 03 16:50:22.046755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.046677 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-model-cache\") pod \"28625d13-a0fa-4bad-919b-3c325c396db2\" (UID: \"28625d13-a0fa-4bad-919b-3c325c396db2\") " Nov 03 16:50:22.046755 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.046693 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-tmp-dir" pod "28625d13-a0fa-4bad-919b-3c325c396db2" (UID: "28625d13-a0fa-4bad-919b-3c325c396db2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:22.047136 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047084 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-model-cache" pod "28625d13-a0fa-4bad-919b-3c325c396db2" (UID: "28625d13-a0fa-4bad-919b-3c325c396db2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:22.047260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047134 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047154 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047168 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047184 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047198 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424b9fdd-1579-4162-827c-e48d0c5af5c9-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047246 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm766\" (UniqueName: \"kubernetes.io/projected/424b9fdd-1579-4162-827c-e48d0c5af5c9-kube-api-access-vm766\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047259 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjhc\" (UniqueName: \"kubernetes.io/projected/8547b3c3-1562-49f9-ab86-cf7a513f9348-kube-api-access-fhjhc\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047275 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047292 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8547b3c3-1562-49f9-ab86-cf7a513f9348-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047307 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424b9fdd-1579-4162-827c-e48d0c5af5c9-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047655 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047322 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsgh7\" (UniqueName: \"kubernetes.io/projected/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4-kube-api-access-hsgh7\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.047878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.047857 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-home" pod "28625d13-a0fa-4bad-919b-3c325c396db2" (UID: "28625d13-a0fa-4bad-919b-3c325c396db2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:22.049509 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.049476 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28625d13-a0fa-4bad-919b-3c325c396db2-kube-api-access-bhhjf" pod "28625d13-a0fa-4bad-919b-3c325c396db2" (UID: "28625d13-a0fa-4bad-919b-3c325c396db2"). InnerVolumeSpecName "kube-api-access-bhhjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:22.049715 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.049696 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28625d13-a0fa-4bad-919b-3c325c396db2-tls-certs" pod "28625d13-a0fa-4bad-919b-3c325c396db2" (UID: "28625d13-a0fa-4bad-919b-3c325c396db2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:22.049850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.049821 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-dshm" pod "28625d13-a0fa-4bad-919b-3c325c396db2" (UID: "28625d13-a0fa-4bad-919b-3c325c396db2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:22.148474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.148400 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.148474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.148425 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28625d13-a0fa-4bad-919b-3c325c396db2-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.148474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.148436 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhhjf\" (UniqueName: \"kubernetes.io/projected/28625d13-a0fa-4bad-919b-3c325c396db2-kube-api-access-bhhjf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.148474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.148445 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.148474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.148453 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28625d13-a0fa-4bad-919b-3c325c396db2-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:22.662270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.662236 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" Nov 03 16:50:22.662747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.662234 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2" event={"ID":"8547b3c3-1562-49f9-ab86-cf7a513f9348","Type":"ContainerDied","Data":"458a139a134c8090528415f081967b8ecbea73b23a9e5f7aab37d54614b48631"} Nov 03 16:50:22.662747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.662384 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="458a139a134c8090528415f081967b8ecbea73b23a9e5f7aab37d54614b48631" Nov 03 16:50:22.663879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.663860 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" Nov 03 16:50:22.664003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.663904 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2" event={"ID":"70f00d68-0ad4-4d1c-83b8-ce2500abb5c4","Type":"ContainerDied","Data":"f18f42618a13abd634b1cbcb9bc447f8ecabbae5e3ccc695a6e56023ddb1a2c2"} Nov 03 16:50:22.664003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.663947 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f18f42618a13abd634b1cbcb9bc447f8ecabbae5e3ccc695a6e56023ddb1a2c2" Nov 03 16:50:22.665614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.665591 5946 generic.go:356] "Generic (PLEG): container finished" podID="28625d13-a0fa-4bad-919b-3c325c396db2" containerID="08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01" exitCode=0 Nov 03 16:50:22.665735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.665624 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" Nov 03 16:50:22.665735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.665684 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" event={"ID":"28625d13-a0fa-4bad-919b-3c325c396db2","Type":"ContainerDied","Data":"08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01"} Nov 03 16:50:22.665735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.665727 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9" event={"ID":"28625d13-a0fa-4bad-919b-3c325c396db2","Type":"ContainerDied","Data":"f400b4be83964f112fd2498e27c6a0a4f1dba9b9b4eb0fbf3462331de5a20d28"} Nov 03 16:50:22.665995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.665750 5946 scope.go:122] "RemoveContainer" containerID="08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01" Nov 03 16:50:22.667525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.667420 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" event={"ID":"424b9fdd-1579-4162-827c-e48d0c5af5c9","Type":"ContainerDied","Data":"19ce76f98d921a488440e7a3efca3d1ade480bc1bdec38b1feb44b9aef01c309"} Nov 03 16:50:22.667525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.667452 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv" Nov 03 16:50:22.676638 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.676622 5946 scope.go:122] "RemoveContainer" containerID="08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01" Nov 03 16:50:22.676907 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:22.676891 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01\": container with ID starting with 08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01 not found: ID does not exist" containerID="08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01" Nov 03 16:50:22.676991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.676922 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01"} err="failed to get container status \"08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01\": rpc error: code = NotFound desc = could not find container \"08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01\": container with ID starting with 08e90ce504d812df11666bacecddb819c6e11938f53510b71a9e6f944458cc01 not found: ID does not exist" Nov 03 16:50:22.676991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.676947 5946 scope.go:122] "RemoveContainer" containerID="32d280cb2c7e356d1e9b25a1ff2678189dda4ac510d38d1d2eb6d03d4f6b2752" Nov 03 16:50:22.842482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.842393 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv"] Nov 03 16:50:22.848989 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:22.848962 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/distinct-2-w5-6ae690-kserve-655676bfd4-2spvv"] Nov 03 16:50:23.019237 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:23.019190 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9"] Nov 03 16:50:23.026234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:23.026167 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6ae690/simulator-w5-6ae690-kserve-69bbccbbcc-qcjx9"] Nov 03 16:50:23.615479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:23.615446 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:23.862423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:23.862376 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:23.931485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:23.931404 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9"] Nov 03 16:50:23.931678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:23.931657 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" containerID="cri-o://02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f" gracePeriod=30 Nov 03 16:50:24.004507 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:24.004474 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" path="/var/lib/kubelet/pods/28625d13-a0fa-4bad-919b-3c325c396db2/volumes" Nov 03 16:50:24.004933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:24.004918 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" path="/var/lib/kubelet/pods/424b9fdd-1579-4162-827c-e48d0c5af5c9/volumes" Nov 03 16:50:24.063383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:24.063350 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:24.257731 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:24.257701 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:25.052694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:25.052655 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:25.246239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:25.246174 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:25.576393 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:25.576347 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:26.103481 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.103448 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:26.577163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577129 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx"] Nov 03 16:50:26.577696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577681 5946 cpu_manager.go:414] "containerMap: removing container" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" Nov 03 16:50:26.577750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577698 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" Nov 03 16:50:26.577750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577716 5946 cpu_manager.go:414] "containerMap: removing container" podUID="70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" containerName="revoke-keys" Nov 03 16:50:26.577750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577721 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" containerName="revoke-keys" Nov 03 16:50:26.577750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577741 5946 cpu_manager.go:414] "containerMap: removing container" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" Nov 03 16:50:26.577750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577747 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" Nov 03 16:50:26.577913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577768 5946 cpu_manager.go:414] "containerMap: removing container" podUID="8547b3c3-1562-49f9-ab86-cf7a513f9348" containerName="revoke-keys" Nov 03 16:50:26.577913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577772 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8547b3c3-1562-49f9-ab86-cf7a513f9348" containerName="revoke-keys" Nov 03 16:50:26.577913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577858 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="8547b3c3-1562-49f9-ab86-cf7a513f9348" containerName="revoke-keys" Nov 03 16:50:26.577913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577871 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="28625d13-a0fa-4bad-919b-3c325c396db2" containerName="main" Nov 03 16:50:26.577913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577879 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" containerName="revoke-keys" Nov 03 16:50:26.577913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.577887 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="424b9fdd-1579-4162-827c-e48d0c5af5c9" containerName="main" Nov 03 16:50:26.582667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.582646 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.645683 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.645648 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7e55c3-openshift-default-dockercfg-fd5b9\"" Nov 03 16:50:26.701012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.700949 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-data\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.712400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.712348 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxx7\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-kube-api-access-4zxx7\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.712812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.712759 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-certs\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.713062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.713031 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f797658-76a2-4569-b387-49b0680f7975-istio-podinfo\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.713284 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.713252 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-credential-socket\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.713486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.713468 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-envoy\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.713704 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.713686 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-socket\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.714008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.713968 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f797658-76a2-4569-b387-49b0680f7975-istiod-ca-cert\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.714340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.714321 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-istio-token\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.717417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.717397 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx"] Nov 03 16:50:26.739074 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.738960 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:26.816182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816145 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-data\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816233 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxx7\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-kube-api-access-4zxx7\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816262 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-certs\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816283 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f797658-76a2-4569-b387-49b0680f7975-istio-podinfo\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816401 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816300 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-credential-socket\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816653 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816629 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-data\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816663 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-envoy\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816691 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-credential-socket\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816715 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-socket\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816805 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f797658-76a2-4569-b387-49b0680f7975-istiod-ca-cert\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.816945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816923 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-istio-token\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.817060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.816640 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-certs\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.817337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.817147 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-socket\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.817789 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.817768 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f797658-76a2-4569-b387-49b0680f7975-istiod-ca-cert\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.819579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.819550 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-envoy\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.819730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.819712 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f797658-76a2-4569-b387-49b0680f7975-istio-podinfo\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.877502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.877402 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxx7\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-kube-api-access-4zxx7\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.877682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.877595 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-istio-token\") pod \"e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:26.900853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:26.900825 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:27.053138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.053109 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:27.351148 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.351115 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx"] Nov 03 16:50:27.351883 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:27.351861 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f797658_76a2_4569_b387_49b0680f7975.slice/crio-67b710931d34e0b5849a5148b68be5600ab76987a2f948b5652414aa898dc8f8 WatchSource:0}: Error finding container 67b710931d34e0b5849a5148b68be5600ab76987a2f948b5652414aa898dc8f8: Status 404 returned error can't find the container with id 67b710931d34e0b5849a5148b68be5600ab76987a2f948b5652414aa898dc8f8 Nov 03 16:50:27.354570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.354534 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:50:27.354688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.354617 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:50:27.354688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.354661 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:50:27.724899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.721788 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" event={"ID":"6f797658-76a2-4569-b387-49b0680f7975","Type":"ContainerStarted","Data":"fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0"} Nov 03 16:50:27.724899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.721871 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" event={"ID":"6f797658-76a2-4569-b387-49b0680f7975","Type":"ContainerStarted","Data":"67b710931d34e0b5849a5148b68be5600ab76987a2f948b5652414aa898dc8f8"} Nov 03 16:50:27.812333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.812295 5946 patch_prober.go:29] interesting pod/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:50:27.812502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.812354 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:50:27.855536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.855468 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podStartSLOduration=1.8554479609999999 podStartE2EDuration="1.855447961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:27.852558154 +0000 UTC m=+2297.973610977" watchObservedRunningTime="2026-11-03 16:50:27.855447961 +0000 UTC m=+2297.976500798" Nov 03 16:50:27.901849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:27.901817 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:28.907394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:28.907361 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:29.202242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.202197 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:50:29.348381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348344 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-certs\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348408 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-credential-socket\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348469 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-data\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348554 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96423ef9-ea37-4e35-ad23-015b1bf7598a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96423ef9-ea37-4e35-ad23-015b1bf7598a-istiod-ca-cert\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348598 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-token\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-token\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348687 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-certs" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:29.348732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348710 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-kube-api-access-jzbcd\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-kube-api-access-jzbcd\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348746 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-podinfo\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348802 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-envoy\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348864 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-socket\") pod \"96423ef9-ea37-4e35-ad23-015b1bf7598a\" (UID: \"96423ef9-ea37-4e35-ad23-015b1bf7598a\") " Nov 03 16:50:29.348907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348870 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-credential-socket" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:29.349155 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.348928 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-data" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:29.349356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.349323 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-socket" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:29.349478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.349404 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.349478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.349426 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.349606 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.349440 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.349830 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.349709 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96423ef9-ea37-4e35-ad23-015b1bf7598a-istiod-ca-cert" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:29.351357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.351331 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-token" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:29.351521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.351504 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-kube-api-access-jzbcd" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "kube-api-access-jzbcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:29.351765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.351741 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-envoy" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:29.351837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.351813 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-podinfo" pod "96423ef9-ea37-4e35-ad23-015b1bf7598a" (UID: "96423ef9-ea37-4e35-ad23-015b1bf7598a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:50:29.451151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.451050 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96423ef9-ea37-4e35-ad23-015b1bf7598a-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.451151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.451084 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.451151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.451100 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzbcd\" (UniqueName: \"kubernetes.io/projected/96423ef9-ea37-4e35-ad23-015b1bf7598a-kube-api-access-jzbcd\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.451151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.451114 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.451151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.451128 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.451151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.451140 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96423ef9-ea37-4e35-ad23-015b1bf7598a-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:29.737383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.737353 5946 generic.go:356] "Generic (PLEG): container finished" podID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerID="02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f" exitCode=0 Nov 03 16:50:29.737558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.737405 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" Nov 03 16:50:29.737558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.737444 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" event={"ID":"96423ef9-ea37-4e35-ad23-015b1bf7598a","Type":"ContainerDied","Data":"02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f"} Nov 03 16:50:29.737558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.737512 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9" event={"ID":"96423ef9-ea37-4e35-ad23-015b1bf7598a","Type":"ContainerDied","Data":"5fa4f9ac52cd0fedb5e02109ffd07f653d37e97f20e8bfe6c868a1a46a7af39e"} Nov 03 16:50:29.737558 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.737536 5946 scope.go:122] "RemoveContainer" containerID="02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f" Nov 03 16:50:29.737843 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.737824 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:29.738837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.738822 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:50:29.748846 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.748827 5946 scope.go:122] "RemoveContainer" containerID="23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a" Nov 03 16:50:29.758368 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.758349 5946 scope.go:122] "RemoveContainer" containerID="02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f" Nov 03 16:50:29.758651 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:29.758632 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f\": container with ID starting with 02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f not found: ID does not exist" containerID="02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f" Nov 03 16:50:29.758707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.758661 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f"} err="failed to get container status \"02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f\": rpc error: code = NotFound desc = could not find container \"02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f\": container with ID starting with 02986df1243580a09098b5ed078ee0fa7798e385d70c165c739570cec7b7e79f not found: ID does not exist" Nov 03 16:50:29.758707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.758677 5946 scope.go:122] "RemoveContainer" containerID="23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a" Nov 03 16:50:29.758924 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:29.758904 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a\": container with ID starting with 23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a not found: ID does not exist" containerID="23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a" Nov 03 16:50:29.758973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.758937 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a"} err="failed to get container status \"23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a\": rpc error: code = NotFound desc = could not find container \"23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a\": container with ID starting with 23ea852e7b09fecd41cd16dbfd59f104b6eec8aca0553daab2f50f396c5c1e8a not found: ID does not exist" Nov 03 16:50:29.773096 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.773070 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:29.922328 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.922293 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9"] Nov 03 16:50:29.926415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:29.926391 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c626cd-openshift-default-6c98f9c4df-4rqq9"] Nov 03 16:50:30.012673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:30.012576 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" path="/var/lib/kubelet/pods/96423ef9-ea37-4e35-ad23-015b1bf7598a/volumes" Nov 03 16:50:30.014224 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:30.014183 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:30.038299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:30.038268 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg"] Nov 03 16:50:30.038486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:30.038467 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" containerID="cri-o://b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713" gracePeriod=30 Nov 03 16:50:30.238164 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:30.238121 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:30.630071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:30.630002 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:30.862030 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:30.861993 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:31.069382 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.069351 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:31.420063 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.419974 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9"] Nov 03 16:50:31.420775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.420750 5946 cpu_manager.go:414] "containerMap: removing container" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" Nov 03 16:50:31.420775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.420777 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" Nov 03 16:50:31.420958 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.420941 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" Nov 03 16:50:31.426107 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.426086 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.495413 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.490710 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ddf492-openshift-default-dockercfg-l8mjw\"" Nov 03 16:50:31.498224 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.498174 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9"] Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.574855 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhnpx\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-kube-api-access-zhnpx\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.574902 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-envoy\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.574932 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-data\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.574966 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istiod-ca-cert\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.574994 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-token\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.575027 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-podinfo\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.575051 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-certs\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.575088 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-socket\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.576265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.575149 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-credential-socket\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.594870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.594781 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:31.676579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676479 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istiod-ca-cert\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676527 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-token\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676556 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-podinfo\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676580 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-certs\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676627 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-socket\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676697 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-credential-socket\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676764 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhnpx\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-kube-api-access-zhnpx\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676813 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-envoy\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.676953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.676845 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-data\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.677310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.677188 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-certs\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.677310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.677249 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-credential-socket\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.677431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.677418 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-socket\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.677498 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.677434 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-data\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.678025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.677999 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istiod-ca-cert\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.679467 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.679443 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-envoy\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.679579 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.679475 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-podinfo\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.742505 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.742472 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhnpx\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-kube-api-access-zhnpx\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.743179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.743138 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-token\") pod \"e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:31.980941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:31.980891 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:32.043115 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.043085 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:32.273013 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.272929 5946 patch_prober.go:29] interesting pod/e2e-api-b-66f145-openshift-default-b8654467-g58fg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:50:32.273420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.273005 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:50:32.488480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.488448 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9"] Nov 03 16:50:32.489713 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:32.489687 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89e95f7_1357_40ff_b001_cf9878dbdf9b.slice/crio-d5a457ef50a537a329b31f787f5d8f0775f50fa874a93da52a0467af930e1d6d WatchSource:0}: Error finding container d5a457ef50a537a329b31f787f5d8f0775f50fa874a93da52a0467af930e1d6d: Status 404 returned error can't find the container with id d5a457ef50a537a329b31f787f5d8f0775f50fa874a93da52a0467af930e1d6d Nov 03 16:50:32.492391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.492362 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:50:32.492475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.492423 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:50:32.492475 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.492451 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:50:32.770931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.770883 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" event={"ID":"a89e95f7-1357-40ff-b001-cf9878dbdf9b","Type":"ContainerStarted","Data":"f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6"} Nov 03 16:50:32.771145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.770949 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" event={"ID":"a89e95f7-1357-40ff-b001-cf9878dbdf9b","Type":"ContainerStarted","Data":"d5a457ef50a537a329b31f787f5d8f0775f50fa874a93da52a0467af930e1d6d"} Nov 03 16:50:32.903671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:32.903564 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podStartSLOduration=1.9035505640000001 podStartE2EDuration="1.903550564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:32.901585457 +0000 UTC m=+2303.022638282" watchObservedRunningTime="2026-11-03 16:50:32.903550564 +0000 UTC m=+2303.024603367" Nov 03 16:50:33.043763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:33.043675 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:33.050614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:33.050443 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:33.777480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:33.777446 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:33.778640 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:33.778614 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:50:33.998173 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:33.998144 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:34.192297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:34.192182 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:34.983868 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:34.983833 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:50:35.360149 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.360122 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:50:35.534724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.534690 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-certs\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.534909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.534738 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-kube-api-access-prr4k\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-kube-api-access-prr4k\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.534909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.534770 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istiod-ca-cert\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.534909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.534794 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-socket\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.535091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.534961 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-podinfo\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.535091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.534975 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-certs" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:35.535091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535054 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-token\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-token\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.535304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535096 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-data\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.535304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535126 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-socket" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:35.535304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535146 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-credential-socket\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.535304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535256 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istiod-ca-cert" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:35.535528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535359 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-envoy\") pod \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\" (UID: \"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e\") " Nov 03 16:50:35.535528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535401 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-data" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:35.535528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535447 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-credential-socket" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:35.535852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535830 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.535966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535857 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.535966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535874 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.535966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535886 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.535966 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.535900 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.537658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.537632 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-token" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:35.537794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.537654 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-kube-api-access-prr4k" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "kube-api-access-prr4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:35.537893 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.537853 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-podinfo" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:50:35.538511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.538492 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-envoy" pod "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" (UID: "23ca2e4c-c89f-4b9a-a774-3bf2777bad9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:35.636743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.636658 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.636743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.636689 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.636743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.636700 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prr4k\" (UniqueName: \"kubernetes.io/projected/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-kube-api-access-prr4k\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.636743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.636710 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:35.669581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.669540 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:35.792441 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.792405 5946 generic.go:356] "Generic (PLEG): container finished" podID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerID="b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713" exitCode=0 Nov 03 16:50:35.792654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.792468 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" Nov 03 16:50:35.792654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.792487 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" event={"ID":"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e","Type":"ContainerDied","Data":"b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713"} Nov 03 16:50:35.792654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.792583 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg" event={"ID":"23ca2e4c-c89f-4b9a-a774-3bf2777bad9e","Type":"ContainerDied","Data":"aefa8650d32cfa988e34da95a7187e7fad6e98860b0e5450958fad9887fe0bda"} Nov 03 16:50:35.792654 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.792617 5946 scope.go:122] "RemoveContainer" containerID="b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713" Nov 03 16:50:35.803952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.803934 5946 scope.go:122] "RemoveContainer" containerID="395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299" Nov 03 16:50:35.814761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.814737 5946 scope.go:122] "RemoveContainer" containerID="b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713" Nov 03 16:50:35.815059 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:35.815041 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713\": container with ID starting with b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713 not found: ID does not exist" containerID="b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713" Nov 03 16:50:35.815105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.815068 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713"} err="failed to get container status \"b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713\": rpc error: code = NotFound desc = could not find container \"b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713\": container with ID starting with b503dfa2fa39a628dae717e366c8e11aa70e6e885cf22809a314a4a433312713 not found: ID does not exist" Nov 03 16:50:35.815105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.815084 5946 scope.go:122] "RemoveContainer" containerID="395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299" Nov 03 16:50:35.815396 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:35.815376 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299\": container with ID starting with 395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299 not found: ID does not exist" containerID="395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299" Nov 03 16:50:35.815455 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.815402 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299"} err="failed to get container status \"395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299\": rpc error: code = NotFound desc = could not find container \"395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299\": container with ID starting with 395fc50f10ea9eef0b96e95e902b1f8afcc23d27e0bf2cd0acead634c4fc3299 not found: ID does not exist" Nov 03 16:50:35.881535 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.881501 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:35.978631 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.978545 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg"] Nov 03 16:50:35.988582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:35.988549 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-66f145-openshift-default-b8654467-g58fg"] Nov 03 16:50:36.004152 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:36.004123 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" path="/var/lib/kubelet/pods/23ca2e4c-c89f-4b9a-a774-3bf2777bad9e/volumes" Nov 03 16:50:36.111984 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:36.111950 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:37.557233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:37.557186 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:39.213040 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.212987 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:39.418563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.418504 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:39.613888 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.613854 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw"] Nov 03 16:50:39.614353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614339 5946 cpu_manager.go:414] "containerMap: removing container" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" Nov 03 16:50:39.614408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614355 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" Nov 03 16:50:39.614408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614375 5946 cpu_manager.go:414] "containerMap: removing container" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" Nov 03 16:50:39.614408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614380 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" Nov 03 16:50:39.614408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614400 5946 cpu_manager.go:414] "containerMap: removing container" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" Nov 03 16:50:39.614408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614406 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" Nov 03 16:50:39.614565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614487 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" Nov 03 16:50:39.614565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614497 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="96423ef9-ea37-4e35-ad23-015b1bf7598a" containerName="istio-proxy" Nov 03 16:50:39.614565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.614506 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ca2e4c-c89f-4b9a-a774-3bf2777bad9e" containerName="istio-proxy" Nov 03 16:50:39.619171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.619154 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:39.624682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.624649 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:39.680318 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.680284 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw"] Nov 03 16:50:39.691851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.691821 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e450076f-d096-468a-8607-cc46d61d1c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:39.692022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.691938 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9spb\" (UniqueName: \"kubernetes.io/projected/e450076f-d096-468a-8607-cc46d61d1c7a-kube-api-access-t9spb\") pod \"payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:39.736234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.735698 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ddf492-dockercfg-lh6wq\"" Nov 03 16:50:39.736234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.735767 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ddf492\"" Nov 03 16:50:39.801279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.799994 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e450076f-d096-468a-8607-cc46d61d1c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:39.801279 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.800120 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9spb\" (UniqueName: \"kubernetes.io/projected/e450076f-d096-468a-8607-cc46d61d1c7a-kube-api-access-t9spb\") pod \"payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:39.802518 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.802441 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e450076f-d096-468a-8607-cc46d61d1c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:39.806603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.806575 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z"] Nov 03 16:50:39.810743 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.810719 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:39.821874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.821852 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:39.829263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.829238 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z"] Nov 03 16:50:39.865584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.865498 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9spb\" (UniqueName: \"kubernetes.io/projected/e450076f-d096-468a-8607-cc46d61d1c7a-kube-api-access-t9spb\") pod \"payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:39.900912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.900870 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36bb1be-b86c-4269-a34c-5d6c2534c102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:39.901067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.901008 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkssh\" (UniqueName: \"kubernetes.io/projected/b36bb1be-b86c-4269-a34c-5d6c2534c102-kube-api-access-hkssh\") pod \"payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:39.935333 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:39.935297 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:40.004912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.002488 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36bb1be-b86c-4269-a34c-5d6c2534c102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:40.004912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.004036 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkssh\" (UniqueName: \"kubernetes.io/projected/b36bb1be-b86c-4269-a34c-5d6c2534c102-kube-api-access-hkssh\") pod \"payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:40.004912 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.004399 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36bb1be-b86c-4269-a34c-5d6c2534c102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:40.065117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.065084 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkssh\" (UniqueName: \"kubernetes.io/projected/b36bb1be-b86c-4269-a34c-5d6c2534c102-kube-api-access-hkssh\") pod \"payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:40.128417 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.128328 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:40.325691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.325660 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw"] Nov 03 16:50:40.327557 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:40.327529 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode450076f_d096_468a_8607_cc46d61d1c7a.slice/crio-f2a7d52d57a603a84f6a1a39010363c9cc8b9f59f630d23bc511f2373b722aca WatchSource:0}: Error finding container f2a7d52d57a603a84f6a1a39010363c9cc8b9f59f630d23bc511f2373b722aca: Status 404 returned error can't find the container with id f2a7d52d57a603a84f6a1a39010363c9cc8b9f59f630d23bc511f2373b722aca Nov 03 16:50:40.510296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.510154 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z"] Nov 03 16:50:40.510818 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:40.510766 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36bb1be_b86c_4269_a34c_5d6c2534c102.slice/crio-8ea16022ee517c14a4fe42b60eb75945ad18e409493e52c0d2b84506d214d155 WatchSource:0}: Error finding container 8ea16022ee517c14a4fe42b60eb75945ad18e409493e52c0d2b84506d214d155: Status 404 returned error can't find the container with id 8ea16022ee517c14a4fe42b60eb75945ad18e409493e52c0d2b84506d214d155 Nov 03 16:50:40.831488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.831434 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" event={"ID":"b36bb1be-b86c-4269-a34c-5d6c2534c102","Type":"ContainerStarted","Data":"fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8"} Nov 03 16:50:40.831710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.831510 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" event={"ID":"b36bb1be-b86c-4269-a34c-5d6c2534c102","Type":"ContainerStarted","Data":"8ea16022ee517c14a4fe42b60eb75945ad18e409493e52c0d2b84506d214d155"} Nov 03 16:50:40.831710 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.831617 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:40.833285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.833254 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" event={"ID":"e450076f-d096-468a-8607-cc46d61d1c7a","Type":"ContainerStarted","Data":"8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456"} Nov 03 16:50:40.833406 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.833304 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" event={"ID":"e450076f-d096-468a-8607-cc46d61d1c7a","Type":"ContainerStarted","Data":"f2a7d52d57a603a84f6a1a39010363c9cc8b9f59f630d23bc511f2373b722aca"} Nov 03 16:50:40.833451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.833409 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:40.955124 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:40.955074 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" podStartSLOduration=1.955059512 podStartE2EDuration="1.955059512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:40.95163289 +0000 UTC m=+2311.072685735" watchObservedRunningTime="2026-11-03 16:50:40.955059512 +0000 UTC m=+2311.076112334" Nov 03 16:50:41.072645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:41.072570 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" podStartSLOduration=2.072549234 podStartE2EDuration="2.072549234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:41.071369508 +0000 UTC m=+2311.192422343" watchObservedRunningTime="2026-11-03 16:50:41.072549234 +0000 UTC m=+2311.193602057" Nov 03 16:50:41.258827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:41.258766 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:41.497052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:41.497016 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:41.690405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:41.690311 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:41.880015 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:41.879979 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:42.584703 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:50:42.584663 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:50:44.834456 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:44.834424 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:45.031988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.031954 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:45.889696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.889655 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt"] Nov 03 16:50:45.893906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.893882 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:45.947011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.946973 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl"] Nov 03 16:50:45.951585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.950936 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:45.954772 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.954743 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt"] Nov 03 16:50:45.973805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.973755 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqk4\" (UniqueName: \"kubernetes.io/projected/2570bbf4-396f-43af-9889-b81fa543d961-kube-api-access-jbqk4\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:45.973805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.973809 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:45.974065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.973953 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:45.974065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.973993 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:45.974256 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.974190 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:45.974347 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.974306 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrqh\" (UniqueName: \"kubernetes.io/projected/74280172-aaf1-450e-958d-e9c1c861d259-kube-api-access-xbrqh\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:45.995070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.995037 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:50:45.995070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:45.995037 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:50:46.000768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.000733 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:50:46.000938 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.000785 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:50:46.012195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.012157 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl"] Nov 03 16:50:46.075869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.075828 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:46.076083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.075908 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrqh\" (UniqueName: \"kubernetes.io/projected/74280172-aaf1-450e-958d-e9c1c861d259-kube-api-access-xbrqh\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:46.076083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.075987 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqk4\" (UniqueName: \"kubernetes.io/projected/2570bbf4-396f-43af-9889-b81fa543d961-kube-api-access-jbqk4\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:46.076083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.076016 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:46.076330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.076124 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:46.076330 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.076199 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:46.076712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.076693 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:46.077473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.077445 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:46.079062 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.079040 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:46.079340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.079314 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:46.136740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.136709 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqk4\" (UniqueName: \"kubernetes.io/projected/2570bbf4-396f-43af-9889-b81fa543d961-kube-api-access-jbqk4\") pod \"maas-api-revoke-sub-8418eb9c-hdxpl\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:46.136901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.136746 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrqh\" (UniqueName: \"kubernetes.io/projected/74280172-aaf1-450e-958d-e9c1c861d259-kube-api-access-xbrqh\") pod \"maas-api-revoke-sub-87d7afab-lmzrt\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:46.142501 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.142423 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:46.210880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.210830 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:46.271645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.270474 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:46.720192 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:46.720155 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2570bbf4_396f_43af_9889_b81fa543d961.slice/crio-1ae7ccf757a408d9fc298c8a0c9fc233a001e88cf74bcd88d1ca6f0de0f34117 WatchSource:0}: Error finding container 1ae7ccf757a408d9fc298c8a0c9fc233a001e88cf74bcd88d1ca6f0de0f34117: Status 404 returned error can't find the container with id 1ae7ccf757a408d9fc298c8a0c9fc233a001e88cf74bcd88d1ca6f0de0f34117 Nov 03 16:50:46.720967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.720941 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl"] Nov 03 16:50:46.782179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.782152 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt"] Nov 03 16:50:46.783160 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:50:46.783136 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74280172_aaf1_450e_958d_e9c1c861d259.slice/crio-9e9bac2d651bb5bb6dbccc3b57c725265eda91447de2a7b905b7e6ba8c0254be WatchSource:0}: Error finding container 9e9bac2d651bb5bb6dbccc3b57c725265eda91447de2a7b905b7e6ba8c0254be: Status 404 returned error can't find the container with id 9e9bac2d651bb5bb6dbccc3b57c725265eda91447de2a7b905b7e6ba8c0254be Nov 03 16:50:46.881000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.880962 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" event={"ID":"74280172-aaf1-450e-958d-e9c1c861d259","Type":"ContainerStarted","Data":"9e9bac2d651bb5bb6dbccc3b57c725265eda91447de2a7b905b7e6ba8c0254be"} Nov 03 16:50:46.882561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:46.882521 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" event={"ID":"2570bbf4-396f-43af-9889-b81fa543d961","Type":"ContainerStarted","Data":"1ae7ccf757a408d9fc298c8a0c9fc233a001e88cf74bcd88d1ca6f0de0f34117"} Nov 03 16:50:47.144163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:47.144133 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:47.760857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:47.760828 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:47.890487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:47.890450 5946 generic.go:356] "Generic (PLEG): container finished" podID="2570bbf4-396f-43af-9889-b81fa543d961" containerID="befabdadc34b132ee44bc9e83c72bc23c835069a7980657b878b0d3c0a4beb94" exitCode=0 Nov 03 16:50:47.890629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:47.890540 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" event={"ID":"2570bbf4-396f-43af-9889-b81fa543d961","Type":"ContainerDied","Data":"befabdadc34b132ee44bc9e83c72bc23c835069a7980657b878b0d3c0a4beb94"} Nov 03 16:50:47.892371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:47.892349 5946 generic.go:356] "Generic (PLEG): container finished" podID="74280172-aaf1-450e-958d-e9c1c861d259" containerID="40e568de03d0edca26ddf2feaf63198c14746b5b49d0b05c359ce1e3034b298a" exitCode=0 Nov 03 16:50:47.892488 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:47.892415 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" event={"ID":"74280172-aaf1-450e-958d-e9c1c861d259","Type":"ContainerDied","Data":"40e568de03d0edca26ddf2feaf63198c14746b5b49d0b05c359ce1e3034b298a"} Nov 03 16:50:48.024639 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:48.024603 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:48.222834 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:48.222792 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:48.371601 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:48.371518 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88"] Nov 03 16:50:48.377254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:48.377225 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-547a38-7cc88"] Nov 03 16:50:48.788733 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:48.788697 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:49.054583 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.054560 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:49.075397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.075370 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:49.152529 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.152496 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2570bbf4-396f-43af-9889-b81fa543d961-kube-api-access-jbqk4\" (UniqueName: \"kubernetes.io/projected/2570bbf4-396f-43af-9889-b81fa543d961-kube-api-access-jbqk4\") pod \"2570bbf4-396f-43af-9889-b81fa543d961\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " Nov 03 16:50:49.152705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.152648 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74280172-aaf1-450e-958d-e9c1c861d259-kube-api-access-xbrqh\" (UniqueName: \"kubernetes.io/projected/74280172-aaf1-450e-958d-e9c1c861d259-kube-api-access-xbrqh\") pod \"74280172-aaf1-450e-958d-e9c1c861d259\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " Nov 03 16:50:49.152705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.152681 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-xks-ca\") pod \"74280172-aaf1-450e-958d-e9c1c861d259\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " Nov 03 16:50:49.152841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.152713 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-xks-ca\") pod \"2570bbf4-396f-43af-9889-b81fa543d961\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " Nov 03 16:50:49.152841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.152759 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-openshift-ca\") pod \"74280172-aaf1-450e-958d-e9c1c861d259\" (UID: \"74280172-aaf1-450e-958d-e9c1c861d259\") " Nov 03 16:50:49.152841 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.152790 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-openshift-ca\") pod \"2570bbf4-396f-43af-9889-b81fa543d961\" (UID: \"2570bbf4-396f-43af-9889-b81fa543d961\") " Nov 03 16:50:49.153458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.153140 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-openshift-ca" pod "74280172-aaf1-450e-958d-e9c1c861d259" (UID: "74280172-aaf1-450e-958d-e9c1c861d259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:49.153458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.153196 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-openshift-ca" pod "2570bbf4-396f-43af-9889-b81fa543d961" (UID: "2570bbf4-396f-43af-9889-b81fa543d961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:49.153458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.153411 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:49.153458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.153433 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:49.155622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.155596 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-xks-ca" pod "2570bbf4-396f-43af-9889-b81fa543d961" (UID: "2570bbf4-396f-43af-9889-b81fa543d961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:49.155696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.155653 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74280172-aaf1-450e-958d-e9c1c861d259-kube-api-access-xbrqh" pod "74280172-aaf1-450e-958d-e9c1c861d259" (UID: "74280172-aaf1-450e-958d-e9c1c861d259"). InnerVolumeSpecName "kube-api-access-xbrqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:49.155696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.155661 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2570bbf4-396f-43af-9889-b81fa543d961-kube-api-access-jbqk4" pod "2570bbf4-396f-43af-9889-b81fa543d961" (UID: "2570bbf4-396f-43af-9889-b81fa543d961"). InnerVolumeSpecName "kube-api-access-jbqk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:49.155696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.155667 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-xks-ca" pod "74280172-aaf1-450e-958d-e9c1c861d259" (UID: "74280172-aaf1-450e-958d-e9c1c861d259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:50:49.254994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.254959 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqk4\" (UniqueName: \"kubernetes.io/projected/2570bbf4-396f-43af-9889-b81fa543d961-kube-api-access-jbqk4\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:49.254994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.254990 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrqh\" (UniqueName: \"kubernetes.io/projected/74280172-aaf1-450e-958d-e9c1c861d259-kube-api-access-xbrqh\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:49.254994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.255001 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74280172-aaf1-450e-958d-e9c1c861d259-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:49.255512 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.255011 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2570bbf4-396f-43af-9889-b81fa543d961-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:50:49.908746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.908710 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" Nov 03 16:50:49.908746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.908734 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt" event={"ID":"74280172-aaf1-450e-958d-e9c1c861d259","Type":"ContainerDied","Data":"9e9bac2d651bb5bb6dbccc3b57c725265eda91447de2a7b905b7e6ba8c0254be"} Nov 03 16:50:49.908980 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.908817 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e9bac2d651bb5bb6dbccc3b57c725265eda91447de2a7b905b7e6ba8c0254be" Nov 03 16:50:49.910296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.910271 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" Nov 03 16:50:49.910412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.910319 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl" event={"ID":"2570bbf4-396f-43af-9889-b81fa543d961","Type":"ContainerDied","Data":"1ae7ccf757a408d9fc298c8a0c9fc233a001e88cf74bcd88d1ca6f0de0f34117"} Nov 03 16:50:49.910412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:49.910355 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae7ccf757a408d9fc298c8a0c9fc233a001e88cf74bcd88d1ca6f0de0f34117" Nov 03 16:50:50.005292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:50.005258 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8a06e3-f2b2-4e3f-9991-7aba2f096636" path="/var/lib/kubelet/pods/6f8a06e3-f2b2-4e3f-9991-7aba2f096636/volumes" Nov 03 16:50:50.399431 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:50.399401 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:51.843682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:51.843655 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 16:50:51.844147 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:51.843721 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 16:50:53.448521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:53.448483 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:55.650120 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:55.650081 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:56.958099 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:56.958057 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:57.161113 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:57.161079 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:57.283741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:57.283710 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d"] Nov 03 16:50:57.283943 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:57.283914 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" containerID="cri-o://d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e" gracePeriod=30 Nov 03 16:50:57.387235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:57.383124 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:58.521320 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:50:58.521289 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:02.548292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.548260 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:51:02.673877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.673762 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-credential-socket\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.673877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.673812 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-certs\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.673900 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-socket\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674168 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674031 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-kube-api-access-kq9lf\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-kube-api-access-kq9lf\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674226 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-socket" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:02.674327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674298 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-podinfo\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674349 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c27c94b-e6be-42e4-b29a-25b216cf8065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27c94b-e6be-42e4-b29a-25b216cf8065-istiod-ca-cert\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674435 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674387 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-token\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-token\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674452 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-credential-socket" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:02.674546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674505 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-data\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674575 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-envoy\") pod \"7c27c94b-e6be-42e4-b29a-25b216cf8065\" (UID: \"7c27c94b-e6be-42e4-b29a-25b216cf8065\") " Nov 03 16:51:02.674651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674634 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-certs" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:02.674952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.674933 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c27c94b-e6be-42e4-b29a-25b216cf8065-istiod-ca-cert" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:02.675048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.675027 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27c94b-e6be-42e4-b29a-25b216cf8065-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.675140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.675124 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.675240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.675142 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-data" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:02.675240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.675146 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.675240 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.675177 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.677414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.677353 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-token" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:02.677414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.677377 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-envoy" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:02.677614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.677438 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-podinfo" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:02.677614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.677603 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-kube-api-access-kq9lf" pod "7c27c94b-e6be-42e4-b29a-25b216cf8065" (UID: "7c27c94b-e6be-42e4-b29a-25b216cf8065"). InnerVolumeSpecName "kube-api-access-kq9lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:02.776724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.776687 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9lf\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-kube-api-access-kq9lf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.776724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.776726 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.777071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.776744 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.777071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.776756 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:02.777071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:02.776769 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27c94b-e6be-42e4-b29a-25b216cf8065-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:03.007766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.007724 5946 generic.go:356] "Generic (PLEG): container finished" podID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerID="d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e" exitCode=0 Nov 03 16:51:03.007918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.007786 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" Nov 03 16:51:03.007918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.007794 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" event={"ID":"7c27c94b-e6be-42e4-b29a-25b216cf8065","Type":"ContainerDied","Data":"d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e"} Nov 03 16:51:03.007918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.007878 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d" event={"ID":"7c27c94b-e6be-42e4-b29a-25b216cf8065","Type":"ContainerDied","Data":"8163c448292bd46b3e91b735e75d385ebf87c44312bffe4656ef56c35b79db65"} Nov 03 16:51:03.007918 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.007908 5946 scope.go:122] "RemoveContainer" containerID="d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e" Nov 03 16:51:03.020788 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.020756 5946 scope.go:122] "RemoveContainer" containerID="ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778" Nov 03 16:51:03.035399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.035377 5946 scope.go:122] "RemoveContainer" containerID="d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e" Nov 03 16:51:03.035764 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:51:03.035738 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e\": container with ID starting with d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e not found: ID does not exist" containerID="d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e" Nov 03 16:51:03.035848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.035776 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e"} err="failed to get container status \"d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e\": rpc error: code = NotFound desc = could not find container \"d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e\": container with ID starting with d1bab19db0e71533f918fc801b4b4549326a9c683ed2f9c62ed31ed47594076e not found: ID does not exist" Nov 03 16:51:03.035848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.035801 5946 scope.go:122] "RemoveContainer" containerID="ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778" Nov 03 16:51:03.036074 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:51:03.036057 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778\": container with ID starting with ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778 not found: ID does not exist" containerID="ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778" Nov 03 16:51:03.036137 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.036080 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778"} err="failed to get container status \"ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778\": rpc error: code = NotFound desc = could not find container \"ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778\": container with ID starting with ba52703ef5001999510f79e998e7b06affc89665423af3f609b66dfcf6dc1778 not found: ID does not exist" Nov 03 16:51:03.182158 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.182118 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d"] Nov 03 16:51:03.231911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.231876 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-mz64d"] Nov 03 16:51:03.673046 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:03.672990 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:04.005461 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:04.005424 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" path="/var/lib/kubelet/pods/7c27c94b-e6be-42e4-b29a-25b216cf8065/volumes" Nov 03 16:51:04.072596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:04.072538 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:04.517126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:04.517091 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:05.050796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.050765 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw"] Nov 03 16:51:05.051334 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051319 5946 cpu_manager.go:414] "containerMap: removing container" podUID="74280172-aaf1-450e-958d-e9c1c861d259" containerName="revoke-keys" Nov 03 16:51:05.051391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051336 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74280172-aaf1-450e-958d-e9c1c861d259" containerName="revoke-keys" Nov 03 16:51:05.051391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051354 5946 cpu_manager.go:414] "containerMap: removing container" podUID="2570bbf4-396f-43af-9889-b81fa543d961" containerName="revoke-keys" Nov 03 16:51:05.051391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051360 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2570bbf4-396f-43af-9889-b81fa543d961" containerName="revoke-keys" Nov 03 16:51:05.051391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051371 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" Nov 03 16:51:05.051391 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051379 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" Nov 03 16:51:05.051561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051401 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" Nov 03 16:51:05.051561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051407 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" Nov 03 16:51:05.051561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051500 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" Nov 03 16:51:05.051561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051515 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="74280172-aaf1-450e-958d-e9c1c861d259" containerName="revoke-keys" Nov 03 16:51:05.051561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.051523 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="2570bbf4-396f-43af-9889-b81fa543d961" containerName="revoke-keys" Nov 03 16:51:05.056790 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.056760 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.107480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.107438 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-credential-socket\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.107480 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.107491 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-data\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.107765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.107711 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-podinfo\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.107833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.107768 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33689f9-d3d1-4959-8112-4b7a2755a4af-istiod-ca-cert\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.109191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.107914 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-socket\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.109191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.107966 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-certs\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.109191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.108061 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-token\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.109191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.108150 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-envoy\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.109191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.108244 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqj6w\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-kube-api-access-qqj6w\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.162220 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.162164 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw"] Nov 03 16:51:05.168235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.168199 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-244c65-openshift-default-dockercfg-knhdz\"" Nov 03 16:51:05.177603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.177573 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:05.212047 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.212007 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-credential-socket\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.212350 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.212315 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-data\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.212633 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.212612 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-podinfo\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.212801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.212783 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33689f9-d3d1-4959-8112-4b7a2755a4af-istiod-ca-cert\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.216779 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.216746 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-socket\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.216939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.216782 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-certs\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.216939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.216841 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-token\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.216939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.216887 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-envoy\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.216939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.216934 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqj6w\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-kube-api-access-qqj6w\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.228917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.228659 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-certs\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.234432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.229834 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33689f9-d3d1-4959-8112-4b7a2755a4af-istiod-ca-cert\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.234432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.230052 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-socket\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.234432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.230986 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-credential-socket\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.234432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.231244 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-data\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.236445 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.235239 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-envoy\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.238642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.238618 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-podinfo\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.284986 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.284960 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqj6w\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-kube-api-access-qqj6w\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.290296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.290269 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-token\") pod \"e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.409400 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.409195 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:05.526807 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.526771 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:05.850724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.850690 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw"] Nov 03 16:51:05.886550 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:51:05.886519 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33689f9_d3d1_4959_8112_4b7a2755a4af.slice/crio-3b2ad3f815af378b61e875d5d3b4705c1c50d573bdd9f99d93f8715198fca7fe WatchSource:0}: Error finding container 3b2ad3f815af378b61e875d5d3b4705c1c50d573bdd9f99d93f8715198fca7fe: Status 404 returned error can't find the container with id 3b2ad3f815af378b61e875d5d3b4705c1c50d573bdd9f99d93f8715198fca7fe Nov 03 16:51:05.889094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.889061 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:05.889220 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.889123 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:05.889220 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:05.889154 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:06.032031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:06.031984 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" event={"ID":"d33689f9-d3d1-4959-8112-4b7a2755a4af","Type":"ContainerStarted","Data":"cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a"} Nov 03 16:51:06.032226 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:06.032059 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" event={"ID":"d33689f9-d3d1-4959-8112-4b7a2755a4af","Type":"ContainerStarted","Data":"3b2ad3f815af378b61e875d5d3b4705c1c50d573bdd9f99d93f8715198fca7fe"} Nov 03 16:51:06.160774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:06.160665 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podStartSLOduration=2.160645961 podStartE2EDuration="2.160645961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:06.155380604 +0000 UTC m=+2336.276433428" watchObservedRunningTime="2026-11-03 16:51:06.160645961 +0000 UTC m=+2336.281698785" Nov 03 16:51:06.409585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:06.409554 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:06.415299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:06.415236 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:07.038853 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:07.038822 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:07.039825 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:07.039805 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:51:08.191060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:08.191027 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:08.649584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:08.649552 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:08.853372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:08.853332 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:09.072090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:09.072057 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:09.294738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:09.294702 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:09.852217 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:09.852184 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:10.057586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:10.057557 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:10.257560 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:10.257524 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh"] Nov 03 16:51:10.308736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:10.308702 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-192564-6dcqh"] Nov 03 16:51:10.317447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:10.317418 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:10.719451 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:10.719367 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:12.003741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:12.003706 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78eae67e-f178-496b-9a7d-f70f553dd5b4" path="/var/lib/kubelet/pods/78eae67e-f178-496b-9a7d-f70f553dd5b4/volumes" Nov 03 16:51:12.584929 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:51:12.584893 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:51:12.929804 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:12.929721 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:13.137004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.136971 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:13.338899 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.338869 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:13.493373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.493342 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r"] Nov 03 16:51:13.493971 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.493955 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c27c94b-e6be-42e4-b29a-25b216cf8065" containerName="istio-proxy" Nov 03 16:51:13.499065 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.499047 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.515652 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.515627 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhns5\" (UniqueName: \"kubernetes.io/projected/301b833f-e81c-4c4b-816c-669b7db045b1-kube-api-access-xhns5\") pod \"payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.515775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.515671 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301b833f-e81c-4c4b-816c-669b7db045b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.556385 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.556360 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r"] Nov 03 16:51:13.604101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.603998 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-244c65-dockercfg-97tgd\"" Nov 03 16:51:13.604101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.604093 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-244c65\"" Nov 03 16:51:13.645229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.645018 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhns5\" (UniqueName: \"kubernetes.io/projected/301b833f-e81c-4c4b-816c-669b7db045b1-kube-api-access-xhns5\") pod \"payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.645229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.645114 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301b833f-e81c-4c4b-816c-669b7db045b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.661268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.646255 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301b833f-e81c-4c4b-816c-669b7db045b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.661268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.660145 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp"] Nov 03 16:51:13.665591 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.665561 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:13.707315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.707287 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhns5\" (UniqueName: \"kubernetes.io/projected/301b833f-e81c-4c4b-816c-669b7db045b1-kube-api-access-xhns5\") pod \"payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.724100 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.724068 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp"] Nov 03 16:51:13.812582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.812550 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:13.848370 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.848339 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e28897-c906-4487-b6c5-c5d588d450c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:13.848581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.848560 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xlqs\" (UniqueName: \"kubernetes.io/projected/51e28897-c906-4487-b6c5-c5d588d450c5-kube-api-access-8xlqs\") pod \"payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:13.949709 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.949672 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlqs\" (UniqueName: \"kubernetes.io/projected/51e28897-c906-4487-b6c5-c5d588d450c5-kube-api-access-8xlqs\") pod \"payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:13.949915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.949737 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e28897-c906-4487-b6c5-c5d588d450c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:13.950570 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:13.950545 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e28897-c906-4487-b6c5-c5d588d450c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:14.010746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:14.010712 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlqs\" (UniqueName: \"kubernetes.io/projected/51e28897-c906-4487-b6c5-c5d588d450c5-kube-api-access-8xlqs\") pod \"payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:14.194189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:14.194164 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r"] Nov 03 16:51:14.195892 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:51:14.195850 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301b833f_e81c_4c4b_816c_669b7db045b1.slice/crio-eac05c8abe0935e7eda1539a157e7e5feb8f80ce49c8c7489a446dbd11faa5a6 WatchSource:0}: Error finding container eac05c8abe0935e7eda1539a157e7e5feb8f80ce49c8c7489a446dbd11faa5a6: Status 404 returned error can't find the container with id eac05c8abe0935e7eda1539a157e7e5feb8f80ce49c8c7489a446dbd11faa5a6 Nov 03 16:51:14.287625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:14.287596 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:14.670095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:14.670069 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp"] Nov 03 16:51:14.671103 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:51:14.671073 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e28897_c906_4487_b6c5_c5d588d450c5.slice/crio-8dc3d3accd40ece45bf29f1496fd9f628b25ced22f9618dcdf8d1d8faf6065f7 WatchSource:0}: Error finding container 8dc3d3accd40ece45bf29f1496fd9f628b25ced22f9618dcdf8d1d8faf6065f7: Status 404 returned error can't find the container with id 8dc3d3accd40ece45bf29f1496fd9f628b25ced22f9618dcdf8d1d8faf6065f7 Nov 03 16:51:15.123833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.123799 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" event={"ID":"51e28897-c906-4487-b6c5-c5d588d450c5","Type":"ContainerStarted","Data":"b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02"} Nov 03 16:51:15.124039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.123859 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" event={"ID":"51e28897-c906-4487-b6c5-c5d588d450c5","Type":"ContainerStarted","Data":"8dc3d3accd40ece45bf29f1496fd9f628b25ced22f9618dcdf8d1d8faf6065f7"} Nov 03 16:51:15.124039 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.123880 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:15.125637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.125600 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" event={"ID":"301b833f-e81c-4c4b-816c-669b7db045b1","Type":"ContainerStarted","Data":"48ec2041b5aa69d03783281a6822497f6821b5501ca457daedec759809330e0a"} Nov 03 16:51:15.125637 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.125642 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" event={"ID":"301b833f-e81c-4c4b-816c-669b7db045b1","Type":"ContainerStarted","Data":"eac05c8abe0935e7eda1539a157e7e5feb8f80ce49c8c7489a446dbd11faa5a6"} Nov 03 16:51:15.125818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.125661 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:15.245486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.245429 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" podStartSLOduration=2.245414324 podStartE2EDuration="2.245414324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:15.244037409 +0000 UTC m=+2345.365090243" watchObservedRunningTime="2026-11-03 16:51:15.245414324 +0000 UTC m=+2345.366467141" Nov 03 16:51:15.269625 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.269471 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:15.365384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:15.365333 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" podStartSLOduration=2.365317337 podStartE2EDuration="2.365317337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:15.363701153 +0000 UTC m=+2345.484753987" watchObservedRunningTime="2026-11-03 16:51:15.365317337 +0000 UTC m=+2345.486370160" Nov 03 16:51:17.096879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:17.096829 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:51:19.408651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:19.408619 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:19.613931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:19.613898 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:19.824783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:19.824747 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:20.267823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:20.267785 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:20.477304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:20.477256 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:20.676424 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:20.676330 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:23.175397 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:23.175365 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:23.369746 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:23.369712 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:25.170070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:25.169958 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:26.135157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:26.135129 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:51:26.135345 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:26.135195 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:51:29.817818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:29.817782 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:30.018714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:30.018678 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:34.670092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:34.670043 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:37.877574 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:37.877542 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:39.374380 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:39.374340 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:42.585360 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:51:42.585323 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:51:43.513328 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:43.513295 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:45.046297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:45.046193 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:45.255066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:45.255024 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:45.456184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:45.456096 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:45.507080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:45.507049 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp"] Nov 03 16:51:45.557967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:45.557936 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6ae690-l2vgp"] Nov 03 16:51:45.925775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:45.925745 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:46.005098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:46.005065 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdbb5ff-b2fc-448c-9766-3086cc9ba55c" path="/var/lib/kubelet/pods/3fdbb5ff-b2fc-448c-9766-3086cc9ba55c/volumes" Nov 03 16:51:47.415962 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:47.415927 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:51:50.546233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:50.543778 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:53.075361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:53.075329 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:53.335176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:53.335082 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:53.489458 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:53.488250 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv"] Nov 03 16:51:53.490381 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:53.490344 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" containerID="cri-o://10ca7d68127047967c54968865d4aa337aeb39658ee1d10198f85f34d8c76867" gracePeriod=30 Nov 03 16:51:53.590361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:53.590267 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:55.974537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:55.974412 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:57.017160 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:57.017125 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:57.216273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:57.216238 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:58.600032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.600004 5946 generic.go:356] "Generic (PLEG): container finished" podID="6f797658-76a2-4569-b387-49b0680f7975" containerID="fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0" exitCode=0 Nov 03 16:51:58.600499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.600080 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" event={"ID":"6f797658-76a2-4569-b387-49b0680f7975","Type":"ContainerDied","Data":"fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0"} Nov 03 16:51:58.600577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.600554 5946 scope.go:122] "RemoveContainer" containerID="fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0" Nov 03 16:51:58.602137 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.602104 5946 generic.go:356] "Generic (PLEG): container finished" podID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerID="f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6" exitCode=0 Nov 03 16:51:58.602252 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.602155 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" event={"ID":"a89e95f7-1357-40ff-b001-cf9878dbdf9b","Type":"ContainerDied","Data":"f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6"} Nov 03 16:51:58.602469 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.602436 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:58.602569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.602528 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:58.602569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.602560 5946 scope.go:122] "RemoveContainer" containerID="f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6" Nov 03 16:51:58.602676 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.602572 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:58.604105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.604079 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:58.604229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.604149 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:58.604229 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.604188 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:51:58.604705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.604609 5946 generic.go:356] "Generic (PLEG): container finished" podID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerID="10ca7d68127047967c54968865d4aa337aeb39658ee1d10198f85f34d8c76867" exitCode=0 Nov 03 16:51:58.604705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.604693 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" event={"ID":"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87","Type":"ContainerDied","Data":"10ca7d68127047967c54968865d4aa337aeb39658ee1d10198f85f34d8c76867"} Nov 03 16:51:58.714453 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.714428 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:51:58.766079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766005 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-envoy\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766073 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-podinfo\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766145 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-certs\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766181 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-socket\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766255 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istiod-ca-cert\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766345 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-kube-api-access-fmmzc\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-kube-api-access-fmmzc\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766402 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-credential-socket\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766498 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-data\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.766604 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.766573 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-token\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-token\") pod \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\" (UID: \"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87\") " Nov 03 16:51:58.767474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.767343 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-certs" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:58.767474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.767415 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-socket" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:58.767747 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.767728 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istiod-ca-cert" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:58.768076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.767939 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-credential-socket" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:58.768294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.768250 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-data" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:58.770566 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.770531 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-kube-api-access-fmmzc" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "kube-api-access-fmmzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:58.770780 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.770756 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-envoy" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:58.773860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.773837 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-podinfo" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:58.775650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.775622 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-token" pod "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" (UID: "e75d8da1-b8d2-449f-bbb6-fa0dd60eff87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:58.868612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868515 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.868612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868558 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.868612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868576 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.868612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868590 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmmzc\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-kube-api-access-fmmzc\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.868612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868605 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.869008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868619 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.869008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868633 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.869008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868646 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:58.869008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:58.868661 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:51:59.613095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.613006 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" Nov 03 16:51:59.613095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.613043 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv" event={"ID":"e75d8da1-b8d2-449f-bbb6-fa0dd60eff87","Type":"ContainerDied","Data":"aa3c0a85b7842f0001d30694ec59db058bfdf77ae5c8d8279a838f45ee8debd7"} Nov 03 16:51:59.613728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.613119 5946 scope.go:122] "RemoveContainer" containerID="10ca7d68127047967c54968865d4aa337aeb39658ee1d10198f85f34d8c76867" Nov 03 16:51:59.641644 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.641615 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" event={"ID":"6f797658-76a2-4569-b387-49b0680f7975","Type":"ContainerStarted","Data":"3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580"} Nov 03 16:51:59.644544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.644518 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" event={"ID":"a89e95f7-1357-40ff-b001-cf9878dbdf9b","Type":"ContainerStarted","Data":"4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4"} Nov 03 16:51:59.742231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.741753 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:51:59.901140 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.901009 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:51:59.902061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.902036 5946 patch_prober.go:29] interesting pod/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 16:51:59.902163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.902083 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 16:51:59.948230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.945268 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv"] Nov 03 16:51:59.960911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:51:59.960874 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-mkxjv"] Nov 03 16:52:00.004511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:00.004484 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" path="/var/lib/kubelet/pods/e75d8da1-b8d2-449f-bbb6-fa0dd60eff87/volumes" Nov 03 16:52:00.056692 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:00.056615 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:52:00.058248 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:00.058189 5946 patch_prober.go:29] interesting pod/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 16:52:00.058686 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:00.058657 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 16:52:00.844233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:00.843758 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:00.901493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:00.901460 5946 patch_prober.go:29] interesting pod/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 16:52:00.901673 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:00.901518 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 16:52:01.046231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:01.045646 5946 patch_prober.go:29] interesting pod/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 16:52:01.046231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:01.045711 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 16:52:01.901556 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:01.901522 5946 patch_prober.go:29] interesting pod/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 16:52:01.901954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:01.901569 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 16:52:02.054049 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:02.054010 5946 patch_prober.go:29] interesting pod/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 16:52:02.054389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:02.054359 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 16:52:02.901331 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:02.901286 5946 patch_prober.go:29] interesting pod/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 16:52:02.901527 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:02.901353 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 16:52:03.059234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:03.058485 5946 patch_prober.go:29] interesting pod/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 16:52:03.059234 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:03.058542 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 16:52:03.778410 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:03.778363 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:52:03.901618 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:03.901586 5946 patch_prober.go:29] interesting pod/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 16:52:03.901795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:03.901641 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 16:52:04.044287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:04.043603 5946 patch_prober.go:29] interesting pod/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 16:52:04.044287 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:04.043717 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 16:52:04.901510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:04.901460 5946 patch_prober.go:29] interesting pod/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 03 16:52:04.901921 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:04.901536 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 03 16:52:05.044025 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:05.043982 5946 patch_prober.go:29] interesting pod/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 16:52:05.044303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:05.044047 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 16:52:05.765076 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:05.765044 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:05.905813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:05.905782 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:52:05.906767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:05.906749 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 16:52:05.974626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:05.974589 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:06.048582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:06.048506 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:52:06.049624 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:06.049605 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 16:52:06.550131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:06.550090 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:10.411195 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:10.411146 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:52:10.417666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:10.417645 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:52:10.680705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:10.680602 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:12.125988 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:12.125956 5946 scope.go:122] "RemoveContainer" containerID="ff1c7fd5f790fa53362d8f2a0cd0c03b21f964e47ea5947442b297cb9d2786c9" Nov 03 16:52:12.137091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:12.137067 5946 scope.go:122] "RemoveContainer" containerID="5a14862a6d54b17bd0ac1e13e2a4057845c09642dc68f9cbe1238d9b0c73b586" Nov 03 16:52:12.585351 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:52:12.585308 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:52:15.497550 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:15.497517 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:20.232024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:20.231950 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:20.983651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:20.983613 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:52:25.068656 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:25.068618 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:28.126164 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:28.126120 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:28.627272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:28.627235 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:29.164231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.164008 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697"] Nov 03 16:52:29.175230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.164793 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" Nov 03 16:52:29.175230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.164818 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" Nov 03 16:52:29.175230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.164997 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75d8da1-b8d2-449f-bbb6-fa0dd60eff87" containerName="istio-proxy" Nov 03 16:52:29.179228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.178027 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.239027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.238994 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-61be78-openshift-default-dockercfg-xpd77\"" Nov 03 16:52:29.289517 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289474 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-socket\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.289714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289546 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-credential-socket\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.289714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289608 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-envoy\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.289714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289631 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-token\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.289714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289661 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115e1a7e-4d10-47f8-b610-16383edd7a78-istiod-ca-cert\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.289714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289710 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-data\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.290012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289752 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-podinfo\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.290012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289820 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9lm\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-kube-api-access-4v9lm\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.290012 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.289864 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-certs\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.292906 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.292871 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697"] Nov 03 16:52:29.326265 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.325097 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:29.391294 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391250 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-certs\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391370 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-socket\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391428 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-credential-socket\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391498 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-envoy\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391522 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-token\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391556 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115e1a7e-4d10-47f8-b610-16383edd7a78-istiod-ca-cert\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391616 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-data\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391711 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391660 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-podinfo\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391754 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-certs\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.391927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.391899 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-credential-socket\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.392029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.392006 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-data\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.392029 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.392015 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9lm\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-kube-api-access-4v9lm\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.392353 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.392327 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-socket\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.392817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.392792 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115e1a7e-4d10-47f8-b610-16383edd7a78-istiod-ca-cert\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.395243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.395188 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-envoy\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.395617 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.395599 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-podinfo\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.455919 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.455834 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9lm\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-kube-api-access-4v9lm\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.458533 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.458502 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-token\") pod \"e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.510227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.503822 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:29.569322 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.569288 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:29.959883 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.959862 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697"] Nov 03 16:52:29.961650 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:52:29.961620 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115e1a7e_4d10_47f8_b610_16383edd7a78.slice/crio-13860c844cbd0ef6def24fc92c22db15b4b7a277e550d5cb20035e8de9461b56 WatchSource:0}: Error finding container 13860c844cbd0ef6def24fc92c22db15b4b7a277e550d5cb20035e8de9461b56: Status 404 returned error can't find the container with id 13860c844cbd0ef6def24fc92c22db15b4b7a277e550d5cb20035e8de9461b56 Nov 03 16:52:29.964311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.964277 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:52:29.964418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.964353 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:52:29.964418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:29.964395 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:52:30.363422 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.363325 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:30.907908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.907874 5946 generic.go:356] "Generic (PLEG): container finished" podID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerID="cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a" exitCode=0 Nov 03 16:52:30.908117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.907940 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" event={"ID":"d33689f9-d3d1-4959-8112-4b7a2755a4af","Type":"ContainerDied","Data":"cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a"} Nov 03 16:52:30.908419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.908400 5946 scope.go:122] "RemoveContainer" containerID="cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a" Nov 03 16:52:30.909765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.909740 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" event={"ID":"115e1a7e-4d10-47f8-b610-16383edd7a78","Type":"ContainerStarted","Data":"60e54da4d0a4b604f41f9d10f08769cb2bd02c275a6cca6545d9bc65753b9b02"} Nov 03 16:52:30.909863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.909787 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" event={"ID":"115e1a7e-4d10-47f8-b610-16383edd7a78","Type":"ContainerStarted","Data":"13860c844cbd0ef6def24fc92c22db15b4b7a277e550d5cb20035e8de9461b56"} Nov 03 16:52:30.909972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.909947 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:52:30.910028 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.910015 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:52:30.910067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:30.910051 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:52:31.258839 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:31.258773 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podStartSLOduration=2.258754462 podStartE2EDuration="2.258754462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:31.254971424 +0000 UTC m=+2421.376024247" watchObservedRunningTime="2026-11-03 16:52:31.258754462 +0000 UTC m=+2421.379807282" Nov 03 16:52:31.504034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:31.504006 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:31.510915 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:31.510845 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:31.922545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:31.922146 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" event={"ID":"d33689f9-d3d1-4959-8112-4b7a2755a4af","Type":"ContainerStarted","Data":"3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0"} Nov 03 16:52:31.922545 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:31.922235 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:31.924724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:31.924672 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:52:32.410003 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:32.409968 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:52:32.411243 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:32.411194 5946 patch_prober.go:29] interesting pod/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 16:52:32.411383 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:32.411268 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 16:52:33.410460 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:33.410425 5946 patch_prober.go:29] interesting pod/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 16:52:33.410862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:33.410479 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 16:52:34.201239 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:34.201167 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:34.410669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:34.410637 5946 patch_prober.go:29] interesting pod/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 16:52:34.411082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:34.410703 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 16:52:34.436241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:34.435854 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:35.409806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:35.409769 5946 patch_prober.go:29] interesting pod/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 16:52:35.409994 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:35.409823 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 16:52:35.582360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:35.582266 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:36.410641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:36.410599 5946 patch_prober.go:29] interesting pod/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 16:52:36.410845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:36.410659 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 16:52:37.038815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:37.038778 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:52:37.415473 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:37.415375 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:52:37.985064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:37.985036 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:52:38.181801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:38.181764 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:38.421611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:38.421534 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:38.684547 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:38.684465 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:39.297835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:39.297796 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:39.576321 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:39.576230 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:40.753860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:40.753821 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:41.129528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.129432 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:41.326421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.326383 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:41.522005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.521966 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd"] Nov 03 16:52:41.525071 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.525049 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.561882 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.561844 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68da1c92-b2af-4617-86e2-292d5194cdba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.562099 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.562074 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54j4b\" (UniqueName: \"kubernetes.io/projected/68da1c92-b2af-4617-86e2-292d5194cdba-kube-api-access-54j4b\") pod \"payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.585260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.585230 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd"] Nov 03 16:52:41.647990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.647957 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-61be78\"" Nov 03 16:52:41.648389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.648371 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-61be78-dockercfg-mbsb9\"" Nov 03 16:52:41.675273 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.675231 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68da1c92-b2af-4617-86e2-292d5194cdba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.675730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.675710 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j4b\" (UniqueName: \"kubernetes.io/projected/68da1c92-b2af-4617-86e2-292d5194cdba-kube-api-access-54j4b\") pod \"payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.676649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.676623 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68da1c92-b2af-4617-86e2-292d5194cdba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.702660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.702622 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh"] Nov 03 16:52:41.705823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.705799 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:41.736244 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.736180 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j4b\" (UniqueName: \"kubernetes.io/projected/68da1c92-b2af-4617-86e2-292d5194cdba-kube-api-access-54j4b\") pod \"payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.771151 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.771113 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh"] Nov 03 16:52:41.777492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.777392 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e201e468-b064-47e2-9e65-6b566718d93f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:41.777689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.777659 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfp9\" (UniqueName: \"kubernetes.io/projected/e201e468-b064-47e2-9e65-6b566718d93f-kube-api-access-csfp9\") pod \"payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:41.837523 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.837487 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:41.878764 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.878724 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e201e468-b064-47e2-9e65-6b566718d93f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:41.878949 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.878906 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfp9\" (UniqueName: \"kubernetes.io/projected/e201e468-b064-47e2-9e65-6b566718d93f-kube-api-access-csfp9\") pod \"payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:41.879823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.879766 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e201e468-b064-47e2-9e65-6b566718d93f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:41.940150 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:41.940117 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfp9\" (UniqueName: \"kubernetes.io/projected/e201e468-b064-47e2-9e65-6b566718d93f-kube-api-access-csfp9\") pod \"payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:42.019004 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:42.018966 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:42.219356 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:42.219328 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd"] Nov 03 16:52:42.221036 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:52:42.221010 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68da1c92_b2af_4617_86e2_292d5194cdba.slice/crio-c01ad1270dbdbc3b11e9aa8ea879d92a57ce47820a4e55153ed9a5507a5878d9 WatchSource:0}: Error finding container c01ad1270dbdbc3b11e9aa8ea879d92a57ce47820a4e55153ed9a5507a5878d9: Status 404 returned error can't find the container with id c01ad1270dbdbc3b11e9aa8ea879d92a57ce47820a4e55153ed9a5507a5878d9 Nov 03 16:52:42.400179 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:42.400156 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh"] Nov 03 16:52:42.401736 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:52:42.401703 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode201e468_b064_47e2_9e65_6b566718d93f.slice/crio-832b2f7d96f4f49753d9be50b37c4ec0ccaf3eae3eb759825a6624e6c327ec80 WatchSource:0}: Error finding container 832b2f7d96f4f49753d9be50b37c4ec0ccaf3eae3eb759825a6624e6c327ec80: Status 404 returned error can't find the container with id 832b2f7d96f4f49753d9be50b37c4ec0ccaf3eae3eb759825a6624e6c327ec80 Nov 03 16:52:42.587585 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:52:42.587497 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:52:43.021553 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.021517 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" event={"ID":"68da1c92-b2af-4617-86e2-292d5194cdba","Type":"ContainerStarted","Data":"abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3"} Nov 03 16:52:43.022055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.021568 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" event={"ID":"68da1c92-b2af-4617-86e2-292d5194cdba","Type":"ContainerStarted","Data":"c01ad1270dbdbc3b11e9aa8ea879d92a57ce47820a4e55153ed9a5507a5878d9"} Nov 03 16:52:43.022055 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.021720 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:43.023272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.023245 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" event={"ID":"e201e468-b064-47e2-9e65-6b566718d93f","Type":"ContainerStarted","Data":"f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be"} Nov 03 16:52:43.023420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.023283 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" event={"ID":"e201e468-b064-47e2-9e65-6b566718d93f","Type":"ContainerStarted","Data":"832b2f7d96f4f49753d9be50b37c4ec0ccaf3eae3eb759825a6624e6c327ec80"} Nov 03 16:52:43.023420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.023317 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:43.143189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.143137 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" podStartSLOduration=2.143121648 podStartE2EDuration="2.143121648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:43.140355901 +0000 UTC m=+2433.261408725" watchObservedRunningTime="2026-11-03 16:52:43.143121648 +0000 UTC m=+2433.264174555" Nov 03 16:52:43.264973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.264912 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" podStartSLOduration=2.26489542 podStartE2EDuration="2.26489542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:43.261943804 +0000 UTC m=+2433.382996660" watchObservedRunningTime="2026-11-03 16:52:43.26489542 +0000 UTC m=+2433.385948261" Nov 03 16:52:43.310149 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.310057 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:43.507497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:43.507463 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:45.879911 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:45.879877 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:46.531184 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:46.531149 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:46.724944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:46.724912 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:46.916960 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:46.916870 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7"] Nov 03 16:52:46.920191 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:46.920167 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:46.980851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:46.980819 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7"] Nov 03 16:52:47.032231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.028495 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-963a2f\"" Nov 03 16:52:47.032231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.028687 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-963a2f-dockercfg-8f9qq\"" Nov 03 16:52:47.054682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.053052 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7488k\" (UniqueName: \"kubernetes.io/projected/1b0ace67-04fd-41f6-898b-e00a64411257-kube-api-access-7488k\") pod \"payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:47.054682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.053159 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0ace67-04fd-41f6-898b-e00a64411257-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:47.154520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.154491 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0ace67-04fd-41f6-898b-e00a64411257-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:47.154708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.154631 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7488k\" (UniqueName: \"kubernetes.io/projected/1b0ace67-04fd-41f6-898b-e00a64411257-kube-api-access-7488k\") pod \"payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:47.155187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.155165 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0ace67-04fd-41f6-898b-e00a64411257-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:47.235756 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.235717 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7488k\" (UniqueName: \"kubernetes.io/projected/1b0ace67-04fd-41f6-898b-e00a64411257-kube-api-access-7488k\") pod \"payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:47.533991 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.533897 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:47.917573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:47.917539 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7"] Nov 03 16:52:47.919058 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:52:47.919031 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0ace67_04fd_41f6_898b_e00a64411257.slice/crio-0f2e4f61b0e971839ab574900a71d99512e184fe8708e42c255745421af0d122 WatchSource:0}: Error finding container 0f2e4f61b0e971839ab574900a71d99512e184fe8708e42c255745421af0d122: Status 404 returned error can't find the container with id 0f2e4f61b0e971839ab574900a71d99512e184fe8708e42c255745421af0d122 Nov 03 16:52:48.092903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:48.092866 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" event={"ID":"1b0ace67-04fd-41f6-898b-e00a64411257","Type":"ContainerStarted","Data":"61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55"} Nov 03 16:52:48.093104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:48.092931 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" event={"ID":"1b0ace67-04fd-41f6-898b-e00a64411257","Type":"ContainerStarted","Data":"0f2e4f61b0e971839ab574900a71d99512e184fe8708e42c255745421af0d122"} Nov 03 16:52:48.093104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:48.092996 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:52:48.216034 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:48.215922 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" podStartSLOduration=2.215908215 podStartE2EDuration="2.215908215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:48.214005032 +0000 UTC m=+2438.335057885" watchObservedRunningTime="2026-11-03 16:52:48.215908215 +0000 UTC m=+2438.336961038" Nov 03 16:52:48.724954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:48.724922 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:48.918744 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:48.918704 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:50.964594 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:50.964550 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:54.033869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:54.033838 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 16:52:54.034482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:54.033902 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 16:52:55.653032 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:55.652992 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:59.103666 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:52:59.103632 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 16:53:00.374086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:00.373995 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:05.049407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:05.049340 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:05.065702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:05.065664 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj"] Nov 03 16:53:05.115336 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:05.115305 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9"] Nov 03 16:53:05.117787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:05.117774 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc577fe-fw6kj"] Nov 03 16:53:05.119982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:05.119961 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ff2e23-f2zb9"] Nov 03 16:53:06.004975 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:06.004945 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b5c8b0-0ffa-4e49-8a43-07f390c29056" path="/var/lib/kubelet/pods/82b5c8b0-0ffa-4e49-8a43-07f390c29056/volumes" Nov 03 16:53:06.005427 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:06.005412 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f057cd7-60d5-4e98-ad80-627e7b06eca4" path="/var/lib/kubelet/pods/9f057cd7-60d5-4e98-ad80-627e7b06eca4/volumes" Nov 03 16:53:09.779371 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:09.779329 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:12.174377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:12.174343 5946 scope.go:122] "RemoveContainer" containerID="c4be8dd63cfeee3a31606c4bcf85b2e933a4bdbd470288e304a84f19c81bad38" Nov 03 16:53:12.184478 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:12.184457 5946 scope.go:122] "RemoveContainer" containerID="8ee0f1ab6ea3e80213c1f3741a76b6361b66e985868f855783fe0787b335ae28" Nov 03 16:53:12.193660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:12.193639 5946 scope.go:122] "RemoveContainer" containerID="e2fe04c17434c3e7f3aee307448ef0442ebda970c51ae43d3755657309a7d973" Nov 03 16:53:12.202824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:12.202800 5946 scope.go:122] "RemoveContainer" containerID="d1a63fe927de5a6c844516126f4f43f0fe153389ec48b5e446617481b153bff9" Nov 03 16:53:12.588746 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:53:12.588710 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:53:15.062089 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:15.061980 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:16.169288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:16.169183 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:19.327105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.327068 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz"] Nov 03 16:53:19.332614 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.332593 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.438315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.438281 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz"] Nov 03 16:53:19.439821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.439788 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3b4d9b\"/\"openshift-service-ca.crt\"" Nov 03 16:53:19.439821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.439788 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3b4d9b\"/\"default-dockercfg-c2rrq\"" Nov 03 16:53:19.440011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.439871 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3b4d9b\"/\"test-model-3b4d9b-kserve-self-signed-certs\"" Nov 03 16:53:19.440011 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.439875 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3b4d9b\"/\"kube-root-ca.crt\"" Nov 03 16:53:19.454519 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.454484 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-dshm\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.454658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.454600 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-model-cache\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.454658 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.454651 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tls-certs\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.454758 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.454712 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhbh\" (UniqueName: \"kubernetes.io/projected/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-kube-api-access-snhbh\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.454809 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.454785 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tmp-dir\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.454869 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.454838 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-home\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.457443 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.457417 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:19.555781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.555746 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-dshm\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.555997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.555815 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-model-cache\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.555997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.555870 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tls-certs\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.555997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.555951 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhbh\" (UniqueName: \"kubernetes.io/projected/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-kube-api-access-snhbh\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.555997 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.555989 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tmp-dir\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.556306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.556009 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-home\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.556375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.556343 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-model-cache\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.556375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.556352 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-home\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.556608 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.556581 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tmp-dir\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.558928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.558883 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-dshm\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.558928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.558917 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tls-certs\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.619922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.619838 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhbh\" (UniqueName: \"kubernetes.io/projected/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-kube-api-access-snhbh\") pod \"test-model-3b4d9b-kserve-6df47dc4ff-wc6mz\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.644737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.644702 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:19.891290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:19.887588 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:20.041742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:20.041716 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz"] Nov 03 16:53:20.042630 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:53:20.042602 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdf45822_d8f5_42e5_ae5c_964f4d0a1fc2.slice/crio-3026ae5aeaa7cc391b0911ada05314764699d2a1e6c7e80c3679857ecbd7af3d WatchSource:0}: Error finding container 3026ae5aeaa7cc391b0911ada05314764699d2a1e6c7e80c3679857ecbd7af3d: Status 404 returned error can't find the container with id 3026ae5aeaa7cc391b0911ada05314764699d2a1e6c7e80c3679857ecbd7af3d Nov 03 16:53:20.330878 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:20.330843 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" event={"ID":"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2","Type":"ContainerStarted","Data":"555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225"} Nov 03 16:53:20.331282 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:20.330885 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" event={"ID":"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2","Type":"ContainerStarted","Data":"3026ae5aeaa7cc391b0911ada05314764699d2a1e6c7e80c3679857ecbd7af3d"} Nov 03 16:53:20.453669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:20.453604 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" podStartSLOduration=1.453588504 podStartE2EDuration="1.453588504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:20.451360205 +0000 UTC m=+2470.572413081" watchObservedRunningTime="2026-11-03 16:53:20.453588504 +0000 UTC m=+2470.574641318" Nov 03 16:53:20.645123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:20.645042 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:24.577271 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:24.577155 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:26.766546 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.766507 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl"] Nov 03 16:53:26.770752 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.770733 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.829340 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.829308 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-244c65\"/\"default-dockercfg-xz2hc\"" Nov 03 16:53:26.866086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.866041 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tls-certs\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.866304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.866172 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-dshm\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.866304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.866222 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-model-cache\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.866304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.866242 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tmp-dir\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.866304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.866258 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whsvp\" (UniqueName: \"kubernetes.io/projected/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-kube-api-access-whsvp\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.866304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.866282 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-home\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.880193 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.880164 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-244c65\"/\"test-model-244c65-kserve-self-signed-certs\"" Nov 03 16:53:26.880685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.880662 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-244c65\"/\"kube-root-ca.crt\"" Nov 03 16:53:26.881268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.881251 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-244c65\"/\"openshift-service-ca.crt\"" Nov 03 16:53:26.883698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.883675 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl"] Nov 03 16:53:26.918173 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.918124 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:26.967378 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967336 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-dshm\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.967600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967388 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-model-cache\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.967600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967411 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tmp-dir\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.967600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967433 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whsvp\" (UniqueName: \"kubernetes.io/projected/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-kube-api-access-whsvp\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.967600 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967464 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-home\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.967847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967616 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tls-certs\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.967910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967869 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-model-cache\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.967967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.967906 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tmp-dir\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.968172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.968144 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-home\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.969898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.969870 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-dshm\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:26.970386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:26.970365 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tls-certs\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:27.032280 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:27.032167 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whsvp\" (UniqueName: \"kubernetes.io/projected/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-kube-api-access-whsvp\") pod \"test-model-244c65-kserve-64c47f776c-b74vl\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:27.084020 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:27.083989 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:27.242492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:27.241815 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:27.432432 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:27.432354 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:27.477912 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:53:27.477877 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0e0f2a2_20d5_4700_81aa_94a4f2003a2e.slice/crio-324e92f33bd84cd7765b8a56fbb52fa025d9f52506b60587551d5e07b6ca28da WatchSource:0}: Error finding container 324e92f33bd84cd7765b8a56fbb52fa025d9f52506b60587551d5e07b6ca28da: Status 404 returned error can't find the container with id 324e92f33bd84cd7765b8a56fbb52fa025d9f52506b60587551d5e07b6ca28da Nov 03 16:53:27.480935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:27.480904 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl"] Nov 03 16:53:28.390056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:28.390021 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" event={"ID":"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e","Type":"ContainerStarted","Data":"2710a53538b554fa26ba5793d60e81b233ed52410a762535a7eeed0aa86910d9"} Nov 03 16:53:28.390499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:28.390065 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" event={"ID":"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e","Type":"ContainerStarted","Data":"324e92f33bd84cd7765b8a56fbb52fa025d9f52506b60587551d5e07b6ca28da"} Nov 03 16:53:28.513171 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:28.513120 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" podStartSLOduration=2.513107009 podStartE2EDuration="2.513107009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:28.51146298 +0000 UTC m=+2478.632515824" watchObservedRunningTime="2026-11-03 16:53:28.513107009 +0000 UTC m=+2478.634159831" Nov 03 16:53:28.703662 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:28.703572 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:53:29.084503 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:29.084467 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:29.255439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:29.255391 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:29.645847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:29.645808 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:29.658402 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:29.658374 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:30.417730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:30.417698 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:53:33.901794 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:33.901743 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:34.093242 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:34.093172 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:34.292922 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:34.292893 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:37.084932 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:37.084889 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:37.097660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:37.097634 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:37.464898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:37.464806 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:53:38.536887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:38.536839 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:38.736573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:38.736505 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:42.586728 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:53:42.586689 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:53:43.447454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:43.447392 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:48.240296 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:48.240258 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:48.433931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:48.433871 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:48.639616 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:48.639460 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:53.287476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:53.287416 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:54.611768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:54.611735 5946 generic.go:356] "Generic (PLEG): container finished" podID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerID="60e54da4d0a4b604f41f9d10f08769cb2bd02c275a6cca6545d9bc65753b9b02" exitCode=0 Nov 03 16:53:54.612299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:54.611808 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" event={"ID":"115e1a7e-4d10-47f8-b610-16383edd7a78","Type":"ContainerDied","Data":"60e54da4d0a4b604f41f9d10f08769cb2bd02c275a6cca6545d9bc65753b9b02"} Nov 03 16:53:54.612299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:54.612247 5946 scope.go:122] "RemoveContainer" containerID="60e54da4d0a4b604f41f9d10f08769cb2bd02c275a6cca6545d9bc65753b9b02" Nov 03 16:53:54.613761 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:54.613735 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:53:54.613833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:54.613803 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:53:54.613873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:54.613835 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 16:53:55.642585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:55.642551 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" event={"ID":"115e1a7e-4d10-47f8-b610-16383edd7a78","Type":"ContainerStarted","Data":"e9f7efc14bc85900a644e8c69038e266922531bf0171ebc6c6265ce3829f0d2f"} Nov 03 16:53:56.504913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:56.504874 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:53:56.505109 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:56.505088 5946 patch_prober.go:29] interesting pod/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.505163 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:56.505134 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 16:53:57.504622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:57.504588 5946 patch_prober.go:29] interesting pod/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.505068 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:57.504651 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 16:53:58.108433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:58.108399 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:58.504408 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:58.504368 5946 patch_prober.go:29] interesting pod/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 16:53:58.504641 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:58.504434 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 16:53:59.505052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:59.505010 5946 patch_prober.go:29] interesting pod/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 16:53:59.505599 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:53:59.505084 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 16:54:00.504770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:00.504739 5946 patch_prober.go:29] interesting pod/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 03 16:54:00.504953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:00.504788 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 03 16:54:01.508902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:01.508873 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:54:01.509421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:01.509403 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:54:01.510024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:01.510004 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 16:54:02.871796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:02.871757 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:05.845385 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:05.845348 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:54:07.583528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:07.583492 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:12.244897 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:12.244863 5946 scope.go:122] "RemoveContainer" containerID="5d478cbd98d35ad08b2a7b5e1ab33735a190c5b6df8d5a61ad06d6a2bc402bd7" Nov 03 16:54:12.258327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:12.258300 5946 scope.go:122] "RemoveContainer" containerID="1a056ae853d52f4672220d0a4b7015361627c2736607f52b75b88627ead90de8" Nov 03 16:54:12.267995 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:12.267971 5946 scope.go:122] "RemoveContainer" containerID="cf2157ccdd32e7be78986d814a69c67915fff8a86d59332d1e0765dbe4c82bda" Nov 03 16:54:12.302874 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:12.302750 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:12.586454 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:54:12.586368 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:54:17.019316 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:17.019264 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:21.797241 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:21.797188 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:24.061233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:24.061186 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk"] Nov 03 16:54:24.112056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:24.112024 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79a8d6b0-kd5wk"] Nov 03 16:54:25.648944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:25.648910 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:26.004675 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:26.004645 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cede107-03a3-40fe-b87f-2b91928d568f" path="/var/lib/kubelet/pods/7cede107-03a3-40fe-b87f-2b91928d568f/volumes" Nov 03 16:54:26.497603 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:26.497567 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:31.195349 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:31.195313 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:35.956650 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:35.956613 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:38.844564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:38.844514 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:40.694847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:40.694812 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:42.585426 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:54:42.585386 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:54:45.483180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:45.480574 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:46.750904 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:46.750873 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:49.315098 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:49.315064 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:50.498516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:50.498474 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:55.175850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:55.175807 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:59.062170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:59.062132 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr"] Nov 03 16:54:59.113104 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:59.113073 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b23a481-k8ksr"] Nov 03 16:54:59.448344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:54:59.448241 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:55:00.013471 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:00.013384 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:00.015860 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:00.015700 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf063e91-8e8a-4ac6-a71c-52a8c333d953" path="/var/lib/kubelet/pods/bf063e91-8e8a-4ac6-a71c-52a8c333d953/volumes" Nov 03 16:55:00.061185 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:00.061143 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4"] Nov 03 16:55:00.110926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:00.110888 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c2c3f8-6v6c4"] Nov 03 16:55:00.959791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:00.959746 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:02.005344 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:02.005309 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e0f140-4cdb-45a5-a934-83afee2ea73e" path="/var/lib/kubelet/pods/c0e0f140-4cdb-45a5-a934-83afee2ea73e/volumes" Nov 03 16:55:05.066364 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:05.066330 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:09.750732 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:09.750698 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:12.319454 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:12.319413 5946 scope.go:122] "RemoveContainer" containerID="7201eafb8266f6feb998aa627afc4691424e3b9eb6e42b69fb27e45a2e0b4615" Nov 03 16:55:12.329730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:12.329707 5946 scope.go:122] "RemoveContainer" containerID="771a27d7495a5722ede14d83125f5ab6db695947c48abdcfeb4fe04bdb32a353" Nov 03 16:55:12.586244 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:55:12.586143 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:55:14.489182 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:14.487950 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:19.318767 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:19.318661 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:22.067712 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:22.067683 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2"] Nov 03 16:55:22.117230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:22.117180 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2"] Nov 03 16:55:22.119646 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:22.119630 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69da120b-ssqx2"] Nov 03 16:55:22.121774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:22.121742 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b964ed13-5b6t2"] Nov 03 16:55:24.010528 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:24.010451 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f00d68-0ad4-4d1c-83b8-ce2500abb5c4" path="/var/lib/kubelet/pods/70f00d68-0ad4-4d1c-83b8-ce2500abb5c4/volumes" Nov 03 16:55:24.011153 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:24.011129 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8547b3c3-1562-49f9-ab86-cf7a513f9348" path="/var/lib/kubelet/pods/8547b3c3-1562-49f9-ab86-cf7a513f9348/volumes" Nov 03 16:55:24.013260 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:24.013187 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:26.863833 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:26.863800 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:26.927043 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:26.927008 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:55:28.751225 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:28.751170 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:33.408060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:33.407893 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:38.100689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:38.100589 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:39.197236 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:39.197155 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:40.693489 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:40.693432 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:42.587659 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:55:42.587621 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:55:42.890285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:42.890149 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:46.379886 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:46.379855 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:47.634201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:47.634160 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:50.062493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:50.062462 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt"] Nov 03 16:55:50.113484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:50.113453 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl"] Nov 03 16:55:50.115818 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:50.115804 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d7afab-lmzrt"] Nov 03 16:55:50.117823 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:50.117803 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8418eb9c-hdxpl"] Nov 03 16:55:52.006037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:52.006001 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2570bbf4-396f-43af-9889-b81fa543d961" path="/var/lib/kubelet/pods/2570bbf4-396f-43af-9889-b81fa543d961/volumes" Nov 03 16:55:52.006544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:52.006528 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74280172-aaf1-450e-958d-e9c1c861d259" path="/var/lib/kubelet/pods/74280172-aaf1-450e-958d-e9c1c861d259/volumes" Nov 03 16:55:52.275894 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:52.275810 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:55.081955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:55.081921 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:55:57.027736 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:57.027697 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:57.553361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:57.553318 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t"] Nov 03 16:55:57.553581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:57.553527 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" containerID="cri-o://c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74" gracePeriod=30 Nov 03 16:55:57.798916 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:55:57.798882 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:01.778389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:01.778348 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:02.822959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.822890 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:56:02.918561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918523 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-data\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.918737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918609 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-token\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-token\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.918737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918651 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istiod-ca-cert\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.918737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918708 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-podinfo\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.918920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918746 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-envoy\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.918920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918802 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-data" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:02.918920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918813 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-socket\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.918920 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918898 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-credential-socket\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.919144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.918974 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-certs\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.919144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.919003 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-kube-api-access-2zsvq\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-kube-api-access-2zsvq\") pod \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\" (UID: \"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10\") " Nov 03 16:56:02.919304 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.919168 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istiod-ca-cert" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:56:02.919395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.919350 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-credential-socket" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:02.919754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.919586 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-certs" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:02.919754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.919448 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:02.919754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.919667 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:02.919754 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.919709 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-socket" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:02.921201 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.921178 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-envoy" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:02.921341 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.921249 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-podinfo" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:56:02.921416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.921386 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-token" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:56:02.921669 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:02.921647 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-kube-api-access-2zsvq" pod "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" (UID: "8d6fe0ae-63e2-4c93-9aca-f532d34e9b10"). InnerVolumeSpecName "kube-api-access-2zsvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:56:03.020739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.020698 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:03.020739 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.020741 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:03.020990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.020757 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:03.020990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.020771 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:03.020990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.020786 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:03.020990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.020799 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:03.020990 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.020812 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zsvq\" (UniqueName: \"kubernetes.io/projected/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10-kube-api-access-2zsvq\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:03.550696 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.550658 5946 generic.go:356] "Generic (PLEG): container finished" podID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerID="c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74" exitCode=0 Nov 03 16:56:03.550885 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.550703 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" Nov 03 16:56:03.550885 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.550707 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" event={"ID":"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10","Type":"ContainerDied","Data":"c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74"} Nov 03 16:56:03.550885 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.550763 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t" event={"ID":"8d6fe0ae-63e2-4c93-9aca-f532d34e9b10","Type":"ContainerDied","Data":"d47c21087665e95c63f9bd9a592ff432cbe8bc211a7de1425d23ffaedd13c9c3"} Nov 03 16:56:03.550885 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.550786 5946 scope.go:122] "RemoveContainer" containerID="c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74" Nov 03 16:56:03.562700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.562679 5946 scope.go:122] "RemoveContainer" containerID="c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74" Nov 03 16:56:03.562996 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:56:03.562977 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74\": container with ID starting with c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74 not found: ID does not exist" containerID="c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74" Nov 03 16:56:03.563051 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.563003 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74"} err="failed to get container status \"c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74\": rpc error: code = NotFound desc = could not find container \"c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74\": container with ID starting with c2c362b1092fa8c37f5d5bc24c571a66c8beee728d850be01c535d38abbcec74 not found: ID does not exist" Nov 03 16:56:03.779881 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.779845 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t"] Nov 03 16:56:03.830844 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:03.830766 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-gjp2t"] Nov 03 16:56:04.005384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:04.005354 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" path="/var/lib/kubelet/pods/8d6fe0ae-63e2-4c93-9aca-f532d34e9b10/volumes" Nov 03 16:56:06.507420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:06.507364 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:11.259144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:11.259105 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:12.366996 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.366958 5946 scope.go:122] "RemoveContainer" containerID="054c6762db8266993ab40a5d3fa80fc56ed6bab758783cdfff99bdb3d4c16581" Nov 03 16:56:12.378812 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.378786 5946 scope.go:122] "RemoveContainer" containerID="73e5d1f1bf36fbf4e8c6b07a2b5531378aee13190f45bd2c6b1bd213f46d255a" Nov 03 16:56:12.389595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.389573 5946 scope.go:122] "RemoveContainer" containerID="1a8f4e494dae4ee300199da31a81ed689a2b17f98d3bf7e65358322fc99ec73c" Nov 03 16:56:12.400805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.400778 5946 scope.go:122] "RemoveContainer" containerID="edde3b0db4c09d8eb38f03ccb2eaa2b2271ccd4acbc95ea89728e61f17f4323f" Nov 03 16:56:12.412373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.412347 5946 scope.go:122] "RemoveContainer" containerID="e5b5fb66c0382c00b3d0c7d15158ea5e7df485b0902b4343a8a3968296d83fb1" Nov 03 16:56:12.421771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.421748 5946 scope.go:122] "RemoveContainer" containerID="5246e2dc30f543c66d43a0bb8dab81a721d4eedeea0d52e3d181ad0829825ef9" Nov 03 16:56:12.431083 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.431061 5946 scope.go:122] "RemoveContainer" containerID="a067650cde49b858a391c59da9a6d8e81202607af9e59f08f2384a7d2917b796" Nov 03 16:56:12.439783 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:12.439761 5946 scope.go:122] "RemoveContainer" containerID="6a63199083c54b220a32e1a286bf8c279734a7bfe57fdf69418591fb2d942e14" Nov 03 16:56:12.586934 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:56:12.586896 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:56:16.096187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:16.096154 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:17.001300 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:17.001268 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn"] Nov 03 16:56:17.001521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:17.001487 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" containerID="cri-o://a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3" gracePeriod=30 Nov 03 16:56:18.734305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:18.734264 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-n7tnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:56:18.734777 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:18.734340 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:56:20.949931 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:20.949899 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:22.262061 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.262037 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:56:22.344940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.344864 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-data\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.344940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.344914 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-token\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.344940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.344935 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-kube-api-access-ggnms\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-kube-api-access-ggnms\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.345253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.344974 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istiod-ca-cert\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.345253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345036 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-socket\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.345253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345088 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-credential-socket\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.345253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345150 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-data" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:22.345253 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345174 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-certs\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.345573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345280 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-envoy\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.345573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345371 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-socket" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:22.345573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345378 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-podinfo\") pod \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\" (UID: \"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8\") " Nov 03 16:56:22.345573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345387 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-certs" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:22.345901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345882 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.345901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345902 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.346048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345911 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.346048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345905 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istiod-ca-cert" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:56:22.346048 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.345972 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-credential-socket" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:22.347735 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.347696 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-kube-api-access-ggnms" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "kube-api-access-ggnms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:56:22.347901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.347874 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-token" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:56:22.348022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.347955 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-envoy" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:22.348022 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.348002 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-podinfo" pod "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" (UID: "1a7e77e0-0571-4236-b6c4-a5a7744ff9b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:56:22.447031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.447002 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.447031 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.447033 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.447312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.447044 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggnms\" (UniqueName: \"kubernetes.io/projected/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-kube-api-access-ggnms\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.447312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.447053 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.447312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.447062 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.447312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.447072 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:56:22.685472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.685251 5946 generic.go:356] "Generic (PLEG): container finished" podID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerID="a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3" exitCode=0 Nov 03 16:56:22.685472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.685319 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" Nov 03 16:56:22.685472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.685396 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" event={"ID":"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8","Type":"ContainerDied","Data":"a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3"} Nov 03 16:56:22.685472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.685458 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn" event={"ID":"1a7e77e0-0571-4236-b6c4-a5a7744ff9b8","Type":"ContainerDied","Data":"a587c64783bcd90ac00a8de32894a573e553073672e33483fe20b4c39be109e4"} Nov 03 16:56:22.685760 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.685486 5946 scope.go:122] "RemoveContainer" containerID="a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3" Nov 03 16:56:22.696389 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.696370 5946 scope.go:122] "RemoveContainer" containerID="a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3" Nov 03 16:56:22.696634 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:56:22.696617 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3\": container with ID starting with a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3 not found: ID does not exist" containerID="a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3" Nov 03 16:56:22.696689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.696642 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3"} err="failed to get container status \"a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3\": rpc error: code = NotFound desc = could not find container \"a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3\": container with ID starting with a2ef7f5387423723d67f1bc65154d8919e970018eab6d16798e4448659a61db3 not found: ID does not exist" Nov 03 16:56:22.858285 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.858251 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn"] Nov 03 16:56:22.909409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:22.909376 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-n7tnn"] Nov 03 16:56:24.004595 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:24.004557 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" path="/var/lib/kubelet/pods/1a7e77e0-0571-4236-b6c4-a5a7744ff9b8/volumes" Nov 03 16:56:25.700944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:25.700911 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:30.461901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:30.461868 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:35.310927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:35.310890 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:39.550176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:39.550132 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:40.047796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:40.047749 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:42.586657 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:56:42.586622 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:56:44.859120 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:44.859075 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:46.919219 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:46.919171 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:56:49.606110 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:49.606073 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:54.352511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:54.352472 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:59.045740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:56:59.045702 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:00.961973 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.961942 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv"] Nov 03 16:57:00.962494 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.962480 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" Nov 03 16:57:00.962541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.962496 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" Nov 03 16:57:00.962541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.962508 5946 cpu_manager.go:414] "containerMap: removing container" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" Nov 03 16:57:00.962541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.962513 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" Nov 03 16:57:00.962645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.962596 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7e77e0-0571-4236-b6c4-a5a7744ff9b8" containerName="istio-proxy" Nov 03 16:57:00.962645 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.962607 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6fe0ae-63e2-4c93-9aca-f532d34e9b10" containerName="istio-proxy" Nov 03 16:57:00.966072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.966056 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:00.989510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.989478 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:00.989660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.989536 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4rj\" (UniqueName: \"kubernetes.io/projected/cb487847-275d-4129-9a99-29439ac16a6f-kube-api-access-ls4rj\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:00.989737 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:00.989721 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.076404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.076372 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:57:01.076404 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.076373 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:57:01.076611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.076412 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:57:01.076611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.076447 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:57:01.079018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.078991 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv"] Nov 03 16:57:01.090714 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.090688 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.090873 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.090818 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4rj\" (UniqueName: \"kubernetes.io/projected/cb487847-275d-4129-9a99-29439ac16a6f-kube-api-access-ls4rj\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.091292 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.091265 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.091463 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.091440 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.093768 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.093744 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.151889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.151862 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4rj\" (UniqueName: \"kubernetes.io/projected/cb487847-275d-4129-9a99-29439ac16a6f-kube-api-access-ls4rj\") pod \"maas-api-revoke-sub-0192f3b7-qc4fv\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.278845 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.278810 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:01.667875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.667848 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv"] Nov 03 16:57:01.669794 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:57:01.669765 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb487847_275d_4129_9a99_29439ac16a6f.slice/crio-35935570e2d52c93e0e796dbadc7d77e2eff0f67a576b79438c530760e09f51c WatchSource:0}: Error finding container 35935570e2d52c93e0e796dbadc7d77e2eff0f67a576b79438c530760e09f51c: Status 404 returned error can't find the container with id 35935570e2d52c93e0e796dbadc7d77e2eff0f67a576b79438c530760e09f51c Nov 03 16:57:01.928974 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:01.928938 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" event={"ID":"cb487847-275d-4129-9a99-29439ac16a6f","Type":"ContainerStarted","Data":"35935570e2d52c93e0e796dbadc7d77e2eff0f67a576b79438c530760e09f51c"} Nov 03 16:57:02.939035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:02.938945 5946 generic.go:356] "Generic (PLEG): container finished" podID="cb487847-275d-4129-9a99-29439ac16a6f" containerID="ebdb5f89b4e458623af369acad408bfeebd91e98f8c730c754620250ea37c34b" exitCode=0 Nov 03 16:57:02.939035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:02.938994 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" event={"ID":"cb487847-275d-4129-9a99-29439ac16a6f","Type":"ContainerDied","Data":"ebdb5f89b4e458623af369acad408bfeebd91e98f8c730c754620250ea37c34b"} Nov 03 16:57:03.894965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:03.894913 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:04.093926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.093897 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:04.136141 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.136108 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-xks-ca\") pod \"cb487847-275d-4129-9a99-29439ac16a6f\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " Nov 03 16:57:04.136335 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.136201 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-openshift-ca\") pod \"cb487847-275d-4129-9a99-29439ac16a6f\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " Nov 03 16:57:04.136394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.136374 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb487847-275d-4129-9a99-29439ac16a6f-kube-api-access-ls4rj\" (UniqueName: \"kubernetes.io/projected/cb487847-275d-4129-9a99-29439ac16a6f-kube-api-access-ls4rj\") pod \"cb487847-275d-4129-9a99-29439ac16a6f\" (UID: \"cb487847-275d-4129-9a99-29439ac16a6f\") " Nov 03 16:57:04.136573 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.136553 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-openshift-ca" pod "cb487847-275d-4129-9a99-29439ac16a6f" (UID: "cb487847-275d-4129-9a99-29439ac16a6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:57:04.136913 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.136890 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:04.138499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.138475 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-xks-ca" pod "cb487847-275d-4129-9a99-29439ac16a6f" (UID: "cb487847-275d-4129-9a99-29439ac16a6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:57:04.138636 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.138618 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb487847-275d-4129-9a99-29439ac16a6f-kube-api-access-ls4rj" pod "cb487847-275d-4129-9a99-29439ac16a6f" (UID: "cb487847-275d-4129-9a99-29439ac16a6f"). InnerVolumeSpecName "kube-api-access-ls4rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:04.238479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.238445 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls4rj\" (UniqueName: \"kubernetes.io/projected/cb487847-275d-4129-9a99-29439ac16a6f-kube-api-access-ls4rj\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:04.238479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.238478 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb487847-275d-4129-9a99-29439ac16a6f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:04.954510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.954479 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" Nov 03 16:57:04.954695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.954516 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv" event={"ID":"cb487847-275d-4129-9a99-29439ac16a6f","Type":"ContainerDied","Data":"35935570e2d52c93e0e796dbadc7d77e2eff0f67a576b79438c530760e09f51c"} Nov 03 16:57:04.954695 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:04.954564 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35935570e2d52c93e0e796dbadc7d77e2eff0f67a576b79438c530760e09f51c" Nov 03 16:57:08.768014 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:08.767972 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:10.457808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:10.457779 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:57:10.465827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:10.465805 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 16:57:12.513326 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:12.513293 5946 scope.go:122] "RemoveContainer" containerID="40e568de03d0edca26ddf2feaf63198c14746b5b49d0b05c359ce1e3034b298a" Nov 03 16:57:12.522947 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:12.522930 5946 scope.go:122] "RemoveContainer" containerID="befabdadc34b132ee44bc9e83c72bc23c835069a7980657b878b0d3c0a4beb94" Nov 03 16:57:12.531552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:12.531534 5946 scope.go:122] "RemoveContainer" containerID="969cce4f0227841f4966c64e8b0eb09cb9fa3b5cd83921514d5942c6ca64500b" Nov 03 16:57:12.539520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:12.539504 5946 scope.go:122] "RemoveContainer" containerID="e2ad5c1936aa24a448e23d09c535d4a1487460f82eaa4d7662e2272c997a0681" Nov 03 16:57:12.587132 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:57:12.587096 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:57:13.566629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:13.566573 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:15.134146 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:15.134107 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz"] Nov 03 16:57:15.220701 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:15.220660 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:16.093384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.093344 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7"] Nov 03 16:57:16.093909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.093894 5946 cpu_manager.go:414] "containerMap: removing container" podUID="cb487847-275d-4129-9a99-29439ac16a6f" containerName="revoke-keys" Nov 03 16:57:16.093963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.093911 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb487847-275d-4129-9a99-29439ac16a6f" containerName="revoke-keys" Nov 03 16:57:16.094038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.094029 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb487847-275d-4129-9a99-29439ac16a6f" containerName="revoke-keys" Nov 03 16:57:16.098875 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.098849 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.158126 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.158095 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7"] Nov 03 16:57:16.182131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.182096 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.182395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.182368 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thnk\" (UniqueName: \"kubernetes.io/projected/a51221a7-c981-4038-aac4-2211ecc22195-kube-api-access-4thnk\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.182541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.182430 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.205567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.205544 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:57:16.206272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.206247 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:57:16.206419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.206391 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:57:16.206419 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.206337 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:57:16.284227 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.284171 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thnk\" (UniqueName: \"kubernetes.io/projected/a51221a7-c981-4038-aac4-2211ecc22195-kube-api-access-4thnk\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.284409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.284253 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.284409 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.284310 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.284902 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.284875 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.286795 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.286769 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.345177 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.345097 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thnk\" (UniqueName: \"kubernetes.io/projected/a51221a7-c981-4038-aac4-2211ecc22195-kube-api-access-4thnk\") pod \"maas-api-revoke-sub-b830e813-2mrw7\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.413651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.413615 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:16.795552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:16.795523 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7"] Nov 03 16:57:16.796972 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:57:16.796948 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51221a7_c981_4038_aac4_2211ecc22195.slice/crio-b975a499060cfc975847c24cb4b2e7da90dbc25c75b16e630763b9a9c96f5bc1 WatchSource:0}: Error finding container b975a499060cfc975847c24cb4b2e7da90dbc25c75b16e630763b9a9c96f5bc1: Status 404 returned error can't find the container with id b975a499060cfc975847c24cb4b2e7da90dbc25c75b16e630763b9a9c96f5bc1 Nov 03 16:57:17.028597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:17.028564 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" event={"ID":"a51221a7-c981-4038-aac4-2211ecc22195","Type":"ContainerStarted","Data":"b975a499060cfc975847c24cb4b2e7da90dbc25c75b16e630763b9a9c96f5bc1"} Nov 03 16:57:18.036502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:18.036469 5946 generic.go:356] "Generic (PLEG): container finished" podID="a51221a7-c981-4038-aac4-2211ecc22195" containerID="b3e763cfc45a8e2d2e0695b9517051d689d296dcd3ab5683182c397fb70e1b55" exitCode=0 Nov 03 16:57:18.036889 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:18.036552 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" event={"ID":"a51221a7-c981-4038-aac4-2211ecc22195","Type":"ContainerDied","Data":"b3e763cfc45a8e2d2e0695b9517051d689d296dcd3ab5683182c397fb70e1b55"} Nov 03 16:57:18.378787 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:18.378754 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:19.197722 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.197697 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:19.322647 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.322550 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51221a7-c981-4038-aac4-2211ecc22195-kube-api-access-4thnk\" (UniqueName: \"kubernetes.io/projected/a51221a7-c981-4038-aac4-2211ecc22195-kube-api-access-4thnk\") pod \"a51221a7-c981-4038-aac4-2211ecc22195\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " Nov 03 16:57:19.322799 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.322646 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-xks-ca\") pod \"a51221a7-c981-4038-aac4-2211ecc22195\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " Nov 03 16:57:19.322799 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.322758 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-openshift-ca\") pod \"a51221a7-c981-4038-aac4-2211ecc22195\" (UID: \"a51221a7-c981-4038-aac4-2211ecc22195\") " Nov 03 16:57:19.323487 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.323457 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-openshift-ca" pod "a51221a7-c981-4038-aac4-2211ecc22195" (UID: "a51221a7-c981-4038-aac4-2211ecc22195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:57:19.325450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.325421 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51221a7-c981-4038-aac4-2211ecc22195-kube-api-access-4thnk" pod "a51221a7-c981-4038-aac4-2211ecc22195" (UID: "a51221a7-c981-4038-aac4-2211ecc22195"). InnerVolumeSpecName "kube-api-access-4thnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:19.325590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.325489 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-xks-ca" pod "a51221a7-c981-4038-aac4-2211ecc22195" (UID: "a51221a7-c981-4038-aac4-2211ecc22195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:57:19.424831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.424799 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thnk\" (UniqueName: \"kubernetes.io/projected/a51221a7-c981-4038-aac4-2211ecc22195-kube-api-access-4thnk\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:19.424831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.424830 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:19.424831 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:19.424841 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51221a7-c981-4038-aac4-2211ecc22195-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:20.050297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:20.050266 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" Nov 03 16:57:20.050297 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:20.050276 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7" event={"ID":"a51221a7-c981-4038-aac4-2211ecc22195","Type":"ContainerDied","Data":"b975a499060cfc975847c24cb4b2e7da90dbc25c75b16e630763b9a9c96f5bc1"} Nov 03 16:57:20.050504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:20.050335 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b975a499060cfc975847c24cb4b2e7da90dbc25c75b16e630763b9a9c96f5bc1" Nov 03 16:57:20.333766 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:20.333679 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:20.569436 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:20.569400 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:23.158770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:23.158730 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:27.886272 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:27.886239 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:28.877852 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:28.877819 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:57:30.156415 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.156346 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" podUID="fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" containerName="main" containerID="cri-o://555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225" gracePeriod=45 Nov 03 16:57:30.427957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.427931 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:57:30.554622 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.554580 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-kube-api-access-snhbh\" (UniqueName: \"kubernetes.io/projected/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-kube-api-access-snhbh\") pod \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " Nov 03 16:57:30.554928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.554645 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tls-certs\") pod \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " Nov 03 16:57:30.554928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.554746 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-dshm\") pod \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " Nov 03 16:57:30.554928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.554793 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-model-cache\") pod \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " Nov 03 16:57:30.554928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.554844 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tmp-dir\") pod \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " Nov 03 16:57:30.554928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.554902 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-home\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-home\") pod \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\" (UID: \"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2\") " Nov 03 16:57:30.556612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.556166 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-model-cache" pod "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" (UID: "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:30.556612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.556575 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tmp-dir" pod "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" (UID: "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:30.558357 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.558331 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-home" pod "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" (UID: "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:30.559125 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.559098 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-kube-api-access-snhbh" pod "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" (UID: "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2"). InnerVolumeSpecName "kube-api-access-snhbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:30.563291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.563263 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-dshm" pod "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" (UID: "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:30.563418 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.563353 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tls-certs" pod "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" (UID: "fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:57:30.656439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.656406 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhbh\" (UniqueName: \"kubernetes.io/projected/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-kube-api-access-snhbh\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:30.656439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.656433 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:30.656439 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.656442 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:30.656660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.656451 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:30.656660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.656460 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:30.656660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:30.656469 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:57:31.129325 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.129285 5946 generic.go:356] "Generic (PLEG): container finished" podID="fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" containerID="555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225" exitCode=0 Nov 03 16:57:31.129557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.129337 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" event={"ID":"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2","Type":"ContainerDied","Data":"555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225"} Nov 03 16:57:31.129557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.129352 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" Nov 03 16:57:31.129557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.129367 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" event={"ID":"fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2","Type":"ContainerDied","Data":"3026ae5aeaa7cc391b0911ada05314764699d2a1e6c7e80c3679857ecbd7af3d"} Nov 03 16:57:31.129557 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.129385 5946 scope.go:122] "RemoveContainer" containerID="555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225" Nov 03 16:57:31.139266 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.139245 5946 scope.go:122] "RemoveContainer" containerID="555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225" Nov 03 16:57:31.139561 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:57:31.139538 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225\": container with ID starting with 555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225 not found: ID does not exist" containerID="555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225" Nov 03 16:57:31.139613 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.139569 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225"} err="failed to get container status \"555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225\": rpc error: code = NotFound desc = could not find container \"555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225\": container with ID starting with 555eda587f7e08013575e187be78aa6cd9aa4fd6185128661b371c284cca5225 not found: ID does not exist" Nov 03 16:57:31.302779 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.302744 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz"] Nov 03 16:57:31.305933 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.305908 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz"] Nov 03 16:57:31.406779 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:31.406691 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3b4d9b/test-model-3b4d9b-kserve-6df47dc4ff-wc6mz" podUID="fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.224:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:57:32.005862 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:32.005829 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" path="/var/lib/kubelet/pods/fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2/volumes" Nov 03 16:57:32.657815 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:32.657781 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:37.420688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:37.420660 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:39.858620 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:39.858587 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:42.169000 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:42.168923 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:42.585506 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:57:42.585460 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:57:46.863769 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:46.863671 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:48.165338 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:48.165302 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:48.408959 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:48.408913 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:51.585099 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:51.585054 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:56.221544 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.221507 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl"] Nov 03 16:57:56.254908 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.254872 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:56.446420 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.446386 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:56.754808 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.754770 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh"] Nov 03 16:57:56.755462 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.755443 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a51221a7-c981-4038-aac4-2211ecc22195" containerName="revoke-keys" Nov 03 16:57:56.755524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.755464 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51221a7-c981-4038-aac4-2211ecc22195" containerName="revoke-keys" Nov 03 16:57:56.755524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.755485 5946 cpu_manager.go:414] "containerMap: removing container" podUID="fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" containerName="main" Nov 03 16:57:56.755524 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.755490 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" containerName="main" Nov 03 16:57:56.755632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.755577 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51221a7-c981-4038-aac4-2211ecc22195" containerName="revoke-keys" Nov 03 16:57:56.755632 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.755589 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf45822-d8f5-42e5-ae5c-964f4d0a1fc2" containerName="main" Nov 03 16:57:56.760108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.760090 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.819181 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.819146 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh"] Nov 03 16:57:56.821038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.821009 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:57:56.821038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.821024 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:57:56.821038 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.821013 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:57:56.821363 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.821345 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:57:56.856377 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.856342 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.856555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.856395 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.856555 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.856538 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pz8\" (UniqueName: \"kubernetes.io/projected/a3e8424e-4d55-459f-a156-8a874245b449-kube-api-access-94pz8\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.958008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.957971 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.958194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.958023 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.958194 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.958066 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pz8\" (UniqueName: \"kubernetes.io/projected/a3e8424e-4d55-459f-a156-8a874245b449-kube-api-access-94pz8\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.958707 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.958685 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:56.960516 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:56.960489 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:57.019972 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:57.019907 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pz8\" (UniqueName: \"kubernetes.io/projected/a3e8424e-4d55-459f-a156-8a874245b449-kube-api-access-94pz8\") pod \"maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:57.073952 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:57.073921 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:57:57.455154 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:57.455120 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh"] Nov 03 16:57:57.457071 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:57:57.457038 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e8424e_4d55_459f_a156_8a874245b449.slice/crio-e998f0a14286be318ba4a622aab09017743db386f341ec76df0860714fb6acbe WatchSource:0}: Error finding container e998f0a14286be318ba4a622aab09017743db386f341ec76df0860714fb6acbe: Status 404 returned error can't find the container with id e998f0a14286be318ba4a622aab09017743db386f341ec76df0860714fb6acbe Nov 03 16:57:58.299702 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:58.299663 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" event={"ID":"a3e8424e-4d55-459f-a156-8a874245b449","Type":"ContainerStarted","Data":"04ca839fd20cbc9639778cb3b6017963ceaef70c060cbf68fd885071088df07d"} Nov 03 16:57:58.299855 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:58.299729 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" event={"ID":"a3e8424e-4d55-459f-a156-8a874245b449","Type":"ContainerStarted","Data":"e998f0a14286be318ba4a622aab09017743db386f341ec76df0860714fb6acbe"} Nov 03 16:57:58.420081 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:58.420028 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" podStartSLOduration=2.42001365 podStartE2EDuration="2.42001365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:57:58.41849331 +0000 UTC m=+2748.539546146" watchObservedRunningTime="2026-11-03 16:57:58.42001365 +0000 UTC m=+2748.541066474" Nov 03 16:57:59.307312 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:59.307276 5946 generic.go:356] "Generic (PLEG): container finished" podID="a3e8424e-4d55-459f-a156-8a874245b449" containerID="04ca839fd20cbc9639778cb3b6017963ceaef70c060cbf68fd885071088df07d" exitCode=0 Nov 03 16:57:59.307837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:59.307355 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" event={"ID":"a3e8424e-4d55-459f-a156-8a874245b449","Type":"ContainerDied","Data":"04ca839fd20cbc9639778cb3b6017963ceaef70c060cbf68fd885071088df07d"} Nov 03 16:57:59.854927 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:57:59.854885 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:58:00.385245 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.382009 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:00.468482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.468457 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:58:00.570717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.570643 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:00.600965 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.600932 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-xks-ca\") pod \"a3e8424e-4d55-459f-a156-8a874245b449\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " Nov 03 16:58:00.601144 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.601071 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-openshift-ca\") pod \"a3e8424e-4d55-459f-a156-8a874245b449\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " Nov 03 16:58:00.601329 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.601307 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e8424e-4d55-459f-a156-8a874245b449-kube-api-access-94pz8\" (UniqueName: \"kubernetes.io/projected/a3e8424e-4d55-459f-a156-8a874245b449-kube-api-access-94pz8\") pod \"a3e8424e-4d55-459f-a156-8a874245b449\" (UID: \"a3e8424e-4d55-459f-a156-8a874245b449\") " Nov 03 16:58:00.601474 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.601433 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-openshift-ca" pod "a3e8424e-4d55-459f-a156-8a874245b449" (UID: "a3e8424e-4d55-459f-a156-8a874245b449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:00.601749 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.601728 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:00.603303 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.603279 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-xks-ca" pod "a3e8424e-4d55-459f-a156-8a874245b449" (UID: "a3e8424e-4d55-459f-a156-8a874245b449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:58:00.603917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.603895 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e8424e-4d55-459f-a156-8a874245b449-kube-api-access-94pz8" pod "a3e8424e-4d55-459f-a156-8a874245b449" (UID: "a3e8424e-4d55-459f-a156-8a874245b449"). InnerVolumeSpecName "kube-api-access-94pz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:00.702642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.702608 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pz8\" (UniqueName: \"kubernetes.io/projected/a3e8424e-4d55-459f-a156-8a874245b449-kube-api-access-94pz8\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:00.702642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:00.702641 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e8424e-4d55-459f-a156-8a874245b449-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:01.321946 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:01.321915 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" Nov 03 16:58:01.322157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:01.321949 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh" event={"ID":"a3e8424e-4d55-459f-a156-8a874245b449","Type":"ContainerDied","Data":"e998f0a14286be318ba4a622aab09017743db386f341ec76df0860714fb6acbe"} Nov 03 16:58:01.322157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:01.322000 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e998f0a14286be318ba4a622aab09017743db386f341ec76df0860714fb6acbe" Nov 03 16:58:01.448249 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:01.448183 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:03.924361 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:03.924317 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:06.177806 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:06.177773 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:08.363817 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:08.363785 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:08.562819 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:08.562789 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:10.948942 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:10.948895 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:11.243164 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.243126 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" podUID="f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" containerName="main" containerID="cri-o://2710a53538b554fa26ba5793d60e81b233ed52410a762535a7eeed0aa86910d9" gracePeriod=45 Nov 03 16:58:11.399067 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.399031 5946 generic.go:356] "Generic (PLEG): container finished" podID="f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" containerID="2710a53538b554fa26ba5793d60e81b233ed52410a762535a7eeed0aa86910d9" exitCode=0 Nov 03 16:58:11.399258 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.399078 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" event={"ID":"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e","Type":"ContainerDied","Data":"2710a53538b554fa26ba5793d60e81b233ed52410a762535a7eeed0aa86910d9"} Nov 03 16:58:11.518070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.518044 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:58:11.637299 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637266 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tls-certs\") pod \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " Nov 03 16:58:11.637521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637343 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-home\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-home\") pod \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " Nov 03 16:58:11.637521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637401 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tmp-dir\") pod \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " Nov 03 16:58:11.637521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637421 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-kube-api-access-whsvp\" (UniqueName: \"kubernetes.io/projected/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-kube-api-access-whsvp\") pod \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " Nov 03 16:58:11.637521 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637501 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-model-cache\") pod \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " Nov 03 16:58:11.637763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637562 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-dshm\") pod \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\" (UID: \"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e\") " Nov 03 16:58:11.637763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637646 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-home" pod "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" (UID: "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:11.637763 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637681 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tmp-dir" pod "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" (UID: "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:11.637909 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.637820 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-model-cache" pod "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" (UID: "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:11.639306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.638087 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:11.639306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.638113 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:11.639306 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.638151 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:11.640537 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.640510 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-kube-api-access-whsvp" pod "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" (UID: "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e"). InnerVolumeSpecName "kube-api-access-whsvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:11.640678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.640654 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tls-certs" pod "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" (UID: "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:58:11.640941 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.640914 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-dshm" pod "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" (UID: "f0e0f2a2-20d5-4700-81aa-94a4f2003a2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:11.739485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.739447 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whsvp\" (UniqueName: \"kubernetes.io/projected/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-kube-api-access-whsvp\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:11.739485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.739480 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:11.739485 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:11.739491 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:12.406688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:12.406604 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" event={"ID":"f0e0f2a2-20d5-4700-81aa-94a4f2003a2e","Type":"ContainerDied","Data":"324e92f33bd84cd7765b8a56fbb52fa025d9f52506b60587551d5e07b6ca28da"} Nov 03 16:58:12.406688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:12.406650 5946 scope.go:122] "RemoveContainer" containerID="2710a53538b554fa26ba5793d60e81b233ed52410a762535a7eeed0aa86910d9" Nov 03 16:58:12.407291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:12.406613 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" Nov 03 16:58:12.454116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:12.454076 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl" podUID="f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:58:12.585821 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:12.585789 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl"] Nov 03 16:58:12.588196 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:58:12.588165 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:58:12.589066 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:12.589045 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-244c65/test-model-244c65-kserve-64c47f776c-b74vl"] Nov 03 16:58:14.004162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:14.004123 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" path="/var/lib/kubelet/pods/f0e0f2a2-20d5-4700-81aa-94a4f2003a2e/volumes" Nov 03 16:58:15.672497 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:15.672390 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:17.983438 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:17.983398 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2"] Nov 03 16:58:17.984086 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:17.983686 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" containerID="cri-o://86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f" gracePeriod=30 Nov 03 16:58:18.891230 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:18.891160 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wrvv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:58:18.891412 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:18.891255 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:58:20.403277 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:20.403234 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:23.234568 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.234545 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:58:23.262811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.262753 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-podinfo\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.262811 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.262788 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-kube-api-access-b5wj2\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-kube-api-access-b5wj2\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.262977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.262845 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-credential-socket\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.262977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.262908 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-data\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.262977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.262945 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-envoy\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.263143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.262991 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-socket\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.263143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263125 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-token\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-token\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.263290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263159 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-certs\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.263290 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263185 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istiod-ca-cert\") pod \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\" (UID: \"b122cee5-df0f-4892-b7b4-af92b6f3afa8\") " Nov 03 16:58:23.263407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263261 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-data" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:23.263407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263276 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-credential-socket" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:23.263407 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263319 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-socket" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:23.263582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263523 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-certs" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:23.263700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263663 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istiod-ca-cert" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:23.263700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263694 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.264052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263715 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.264052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263725 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.264052 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.263734 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.265421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.265396 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-podinfo" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:58:23.265513 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.265465 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-token" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:23.265685 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.265662 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-envoy" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:23.265742 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.265694 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-kube-api-access-b5wj2" pod "b122cee5-df0f-4892-b7b4-af92b6f3afa8" (UID: "b122cee5-df0f-4892-b7b4-af92b6f3afa8"). InnerVolumeSpecName "kube-api-access-b5wj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:23.364425 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.364398 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.364425 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.364423 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.364425 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.364432 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.364665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.364440 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122cee5-df0f-4892-b7b4-af92b6f3afa8-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.364665 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.364449 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5wj2\" (UniqueName: \"kubernetes.io/projected/b122cee5-df0f-4892-b7b4-af92b6f3afa8-kube-api-access-b5wj2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:23.483449 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.483417 5946 generic.go:356] "Generic (PLEG): container finished" podID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerID="86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f" exitCode=0 Nov 03 16:58:23.483651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.483473 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" Nov 03 16:58:23.483651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.483482 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" event={"ID":"b122cee5-df0f-4892-b7b4-af92b6f3afa8","Type":"ContainerDied","Data":"86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f"} Nov 03 16:58:23.483651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.483524 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2" event={"ID":"b122cee5-df0f-4892-b7b4-af92b6f3afa8","Type":"ContainerDied","Data":"c2df6729fc9d42017c5439ab35a5c6a7090a69c8dfd39fd6d0da7fa6ec78a0ec"} Nov 03 16:58:23.483651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.483544 5946 scope.go:122] "RemoveContainer" containerID="86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f" Nov 03 16:58:23.494880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.494861 5946 scope.go:122] "RemoveContainer" containerID="86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f" Nov 03 16:58:23.495114 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:58:23.495099 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f\": container with ID starting with 86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f not found: ID does not exist" containerID="86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f" Nov 03 16:58:23.495167 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.495124 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f"} err="failed to get container status \"86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f\": rpc error: code = NotFound desc = could not find container \"86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f\": container with ID starting with 86d6bae16da0a3916c30f38309af0a098b35c1180262eab667545bdb1799664f not found: ID does not exist" Nov 03 16:58:23.657327 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.657244 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2"] Nov 03 16:58:23.663060 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:23.663034 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wrvv2"] Nov 03 16:58:24.003977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:24.003943 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" path="/var/lib/kubelet/pods/b122cee5-df0f-4892-b7b4-af92b6f3afa8/volumes" Nov 03 16:58:24.541360 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:24.541326 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:25.988985 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:25.988941 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:26.265264 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:26.265163 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:26.474581 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:26.474541 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:26.645561 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:26.645467 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh"] Nov 03 16:58:26.695734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:26.695696 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b4d9b-sbqxh"] Nov 03 16:58:27.078687 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:27.078655 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:28.005131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:28.005098 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e8424e-4d55-459f-a156-8a874245b449" path="/var/lib/kubelet/pods/a3e8424e-4d55-459f-a156-8a874245b449/volumes" Nov 03 16:58:31.265187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:31.265122 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:34.560678 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:34.560651 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:34.755721 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:34.755691 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:34.941392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:34.941299 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:35.755611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.755574 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb"] Nov 03 16:58:35.756072 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756058 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a3e8424e-4d55-459f-a156-8a874245b449" containerName="revoke-keys" Nov 03 16:58:35.756117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756072 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e8424e-4d55-459f-a156-8a874245b449" containerName="revoke-keys" Nov 03 16:58:35.756117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756102 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" containerName="main" Nov 03 16:58:35.756117 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756107 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" containerName="main" Nov 03 16:58:35.756254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756135 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" Nov 03 16:58:35.756254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756140 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" Nov 03 16:58:35.756254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756231 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e0f2a2-20d5-4700-81aa-94a4f2003a2e" containerName="main" Nov 03 16:58:35.756254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756245 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b122cee5-df0f-4892-b7b4-af92b6f3afa8" containerName="istio-proxy" Nov 03 16:58:35.756254 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.756251 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e8424e-4d55-459f-a156-8a874245b449" containerName="revoke-keys" Nov 03 16:58:35.759621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.759605 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:35.818691 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.818658 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb"] Nov 03 16:58:35.866824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.866785 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:58:35.866824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.866798 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:58:35.866824 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.866830 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:58:35.867082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.866798 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:58:35.908611 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.908581 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48skc\" (UniqueName: \"kubernetes.io/projected/41386308-3a42-4da4-88ca-0d2901d13872-kube-api-access-48skc\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:35.908775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.908649 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:35.908842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:35.908815 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.009470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.009395 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.009470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.009467 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.009698 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.009535 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48skc\" (UniqueName: \"kubernetes.io/projected/41386308-3a42-4da4-88ca-0d2901d13872-kube-api-access-48skc\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.010087 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.010068 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.011963 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.011943 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.031866 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.031800 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:36.069847 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.069820 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48skc\" (UniqueName: \"kubernetes.io/projected/41386308-3a42-4da4-88ca-0d2901d13872-kube-api-access-48skc\") pod \"maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.073647 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.073622 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:36.454421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.454394 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb"] Nov 03 16:58:36.456668 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:58:36.456640 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41386308_3a42_4da4_88ca_0d2901d13872.slice/crio-1d21d369c21a45e26d2385cc5e24aca8183551c548b07f670438acd4e57f5640 WatchSource:0}: Error finding container 1d21d369c21a45e26d2385cc5e24aca8183551c548b07f670438acd4e57f5640: Status 404 returned error can't find the container with id 1d21d369c21a45e26d2385cc5e24aca8183551c548b07f670438acd4e57f5640 Nov 03 16:58:36.568145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:36.568108 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" event={"ID":"41386308-3a42-4da4-88ca-0d2901d13872","Type":"ContainerStarted","Data":"1d21d369c21a45e26d2385cc5e24aca8183551c548b07f670438acd4e57f5640"} Nov 03 16:58:37.575448 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:37.575409 5946 generic.go:356] "Generic (PLEG): container finished" podID="41386308-3a42-4da4-88ca-0d2901d13872" containerID="c22b30ce6e4c29ebd110961082a5a78e2db7be2d32fb29b793e8b13ad94bb1d9" exitCode=0 Nov 03 16:58:37.575887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:37.575490 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" event={"ID":"41386308-3a42-4da4-88ca-0d2901d13872","Type":"ContainerDied","Data":"c22b30ce6e4c29ebd110961082a5a78e2db7be2d32fb29b793e8b13ad94bb1d9"} Nov 03 16:58:38.739796 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.739773 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:38.844585 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.844501 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41386308-3a42-4da4-88ca-0d2901d13872-kube-api-access-48skc\" (UniqueName: \"kubernetes.io/projected/41386308-3a42-4da4-88ca-0d2901d13872-kube-api-access-48skc\") pod \"41386308-3a42-4da4-88ca-0d2901d13872\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " Nov 03 16:58:38.844741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.844599 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-xks-ca\") pod \"41386308-3a42-4da4-88ca-0d2901d13872\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " Nov 03 16:58:38.844741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.844633 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-openshift-ca\") pod \"41386308-3a42-4da4-88ca-0d2901d13872\" (UID: \"41386308-3a42-4da4-88ca-0d2901d13872\") " Nov 03 16:58:38.845035 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.845002 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-openshift-ca" pod "41386308-3a42-4da4-88ca-0d2901d13872" (UID: "41386308-3a42-4da4-88ca-0d2901d13872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:38.846850 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.846822 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-xks-ca" pod "41386308-3a42-4da4-88ca-0d2901d13872" (UID: "41386308-3a42-4da4-88ca-0d2901d13872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:58:38.846977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.846933 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41386308-3a42-4da4-88ca-0d2901d13872-kube-api-access-48skc" pod "41386308-3a42-4da4-88ca-0d2901d13872" (UID: "41386308-3a42-4da4-88ca-0d2901d13872"). InnerVolumeSpecName "kube-api-access-48skc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:38.945765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.945730 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:38.945765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.945760 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41386308-3a42-4da4-88ca-0d2901d13872-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:38.945765 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:38.945770 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48skc\" (UniqueName: \"kubernetes.io/projected/41386308-3a42-4da4-88ca-0d2901d13872-kube-api-access-48skc\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:39.592270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:39.592244 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" Nov 03 16:58:39.592270 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:39.592254 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb" event={"ID":"41386308-3a42-4da4-88ca-0d2901d13872","Type":"ContainerDied","Data":"1d21d369c21a45e26d2385cc5e24aca8183551c548b07f670438acd4e57f5640"} Nov 03 16:58:39.592504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:39.592303 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d21d369c21a45e26d2385cc5e24aca8183551c548b07f670438acd4e57f5640" Nov 03 16:58:40.182940 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:40.182902 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:40.634134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:40.634093 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:58:41.037156 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:41.037122 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:42.585332 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:58:42.585287 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:58:44.819386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.819346 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh"] Nov 03 16:58:44.819914 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.819902 5946 cpu_manager.go:414] "containerMap: removing container" podUID="41386308-3a42-4da4-88ca-0d2901d13872" containerName="revoke-keys" Nov 03 16:58:44.819979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.819916 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41386308-3a42-4da4-88ca-0d2901d13872" containerName="revoke-keys" Nov 03 16:58:44.820037 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.820010 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="41386308-3a42-4da4-88ca-0d2901d13872" containerName="revoke-keys" Nov 03 16:58:44.823801 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.823785 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:44.882626 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.882589 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh"] Nov 03 16:58:44.930138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.930096 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:58:44.930138 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.930105 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:58:44.930384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.930168 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:58:44.930384 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:44.930244 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:58:45.015183 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.015145 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.015405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.015254 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.015405 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.015371 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpl44\" (UniqueName: \"kubernetes.io/projected/358c779a-f372-4d84-948a-262e032e6dd9-kube-api-access-bpl44\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.116901 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.116815 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpl44\" (UniqueName: \"kubernetes.io/projected/358c779a-f372-4d84-948a-262e032e6dd9-kube-api-access-bpl44\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.117057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.116944 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.117057 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.117009 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.117649 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.117624 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.119569 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.119547 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.177730 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.177691 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpl44\" (UniqueName: \"kubernetes.io/projected/358c779a-f372-4d84-948a-262e032e6dd9-kube-api-access-bpl44\") pod \"maas-api-revoke-sub-095511ef-5fbvh\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.437499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.437412 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:45.836118 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.836089 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh"] Nov 03 16:58:45.837721 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:58:45.837697 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358c779a_f372_4d84_948a_262e032e6dd9.slice/crio-85bd33b10df7278cafbbc4268b5163c288a3d86afdef29ed03e87b513053f9e3 WatchSource:0}: Error finding container 85bd33b10df7278cafbbc4268b5163c288a3d86afdef29ed03e87b513053f9e3: Status 404 returned error can't find the container with id 85bd33b10df7278cafbbc4268b5163c288a3d86afdef29ed03e87b513053f9e3 Nov 03 16:58:45.856157 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:45.856110 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:46.635935 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:46.635893 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" event={"ID":"358c779a-f372-4d84-948a-262e032e6dd9","Type":"ContainerStarted","Data":"2ef0249e7071ac7a6d54680bec190e24cf48d320aa29e0e7bd9b8ecf776b6ed2"} Nov 03 16:58:46.636116 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:46.635949 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" event={"ID":"358c779a-f372-4d84-948a-262e032e6dd9","Type":"ContainerStarted","Data":"85bd33b10df7278cafbbc4268b5163c288a3d86afdef29ed03e87b513053f9e3"} Nov 03 16:58:46.760268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:46.760188 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" podStartSLOduration=2.760176252 podStartE2EDuration="2.760176252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:58:46.756012838 +0000 UTC m=+2796.877065661" watchObservedRunningTime="2026-11-03 16:58:46.760176252 +0000 UTC m=+2796.881229075" Nov 03 16:58:46.810122 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:46.810093 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp"] Nov 03 16:58:46.890482 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:46.890398 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r"] Nov 03 16:58:47.219582 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:47.219499 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:47.420131 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:47.420098 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:47.645145 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:47.645099 5946 generic.go:356] "Generic (PLEG): container finished" podID="358c779a-f372-4d84-948a-262e032e6dd9" containerID="2ef0249e7071ac7a6d54680bec190e24cf48d320aa29e0e7bd9b8ecf776b6ed2" exitCode=0 Nov 03 16:58:47.645359 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:47.645311 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" event={"ID":"358c779a-f372-4d84-948a-262e032e6dd9","Type":"ContainerDied","Data":"2ef0249e7071ac7a6d54680bec190e24cf48d320aa29e0e7bd9b8ecf776b6ed2"} Nov 03 16:58:48.789813 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.789787 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:48.861082 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.861053 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358c779a-f372-4d84-948a-262e032e6dd9-kube-api-access-bpl44\" (UniqueName: \"kubernetes.io/projected/358c779a-f372-4d84-948a-262e032e6dd9-kube-api-access-bpl44\") pod \"358c779a-f372-4d84-948a-262e032e6dd9\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " Nov 03 16:58:48.861317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.861139 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-openshift-ca\") pod \"358c779a-f372-4d84-948a-262e032e6dd9\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " Nov 03 16:58:48.861317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.861162 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-xks-ca\") pod \"358c779a-f372-4d84-948a-262e032e6dd9\" (UID: \"358c779a-f372-4d84-948a-262e032e6dd9\") " Nov 03 16:58:48.861571 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.861544 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-openshift-ca" pod "358c779a-f372-4d84-948a-262e032e6dd9" (UID: "358c779a-f372-4d84-948a-262e032e6dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:48.861725 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.861711 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:48.863450 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.863426 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-xks-ca" pod "358c779a-f372-4d84-948a-262e032e6dd9" (UID: "358c779a-f372-4d84-948a-262e032e6dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:58:48.863588 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.863566 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358c779a-f372-4d84-948a-262e032e6dd9-kube-api-access-bpl44" pod "358c779a-f372-4d84-948a-262e032e6dd9" (UID: "358c779a-f372-4d84-948a-262e032e6dd9"). InnerVolumeSpecName "kube-api-access-bpl44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:48.962955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.962875 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpl44\" (UniqueName: \"kubernetes.io/projected/358c779a-f372-4d84-948a-262e032e6dd9-kube-api-access-bpl44\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:48.962955 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:48.962903 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358c779a-f372-4d84-948a-262e032e6dd9-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:49.659434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:49.659397 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" event={"ID":"358c779a-f372-4d84-948a-262e032e6dd9","Type":"ContainerDied","Data":"85bd33b10df7278cafbbc4268b5163c288a3d86afdef29ed03e87b513053f9e3"} Nov 03 16:58:49.659434 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:49.659423 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh" Nov 03 16:58:49.659651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:49.659455 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85bd33b10df7278cafbbc4268b5163c288a3d86afdef29ed03e87b513053f9e3" Nov 03 16:58:50.647101 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:50.647071 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:51.810510 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:51.810469 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" podUID="51e28897-c906-4487-b6c5-c5d588d450c5" containerName="payload-processing" containerID="cri-o://b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02" gracePeriod=25 Nov 03 16:58:51.890851 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:51.890809 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" podUID="301b833f-e81c-4c4b-816c-669b7db045b1" containerName="payload-pre-processing" containerID="cri-o://48ec2041b5aa69d03783281a6822497f6821b5501ca457daedec759809330e0a" gracePeriod=25 Nov 03 16:58:52.068070 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.068004 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:58:52.096337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.096311 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e28897-c906-4487-b6c5-c5d588d450c5-kube-api-access-8xlqs\" (UniqueName: \"kubernetes.io/projected/51e28897-c906-4487-b6c5-c5d588d450c5-kube-api-access-8xlqs\") pod \"51e28897-c906-4487-b6c5-c5d588d450c5\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " Nov 03 16:58:52.096499 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.096410 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e28897-c906-4487-b6c5-c5d588d450c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e28897-c906-4487-b6c5-c5d588d450c5-plugins-config-volume\") pod \"51e28897-c906-4487-b6c5-c5d588d450c5\" (UID: \"51e28897-c906-4487-b6c5-c5d588d450c5\") " Nov 03 16:58:52.096805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.096773 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e28897-c906-4487-b6c5-c5d588d450c5-plugins-config-volume" pod "51e28897-c906-4487-b6c5-c5d588d450c5" (UID: "51e28897-c906-4487-b6c5-c5d588d450c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:52.098724 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.098704 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e28897-c906-4487-b6c5-c5d588d450c5-kube-api-access-8xlqs" pod "51e28897-c906-4487-b6c5-c5d588d450c5" (UID: "51e28897-c906-4487-b6c5-c5d588d450c5"). InnerVolumeSpecName "kube-api-access-8xlqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:52.197642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.197612 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xlqs\" (UniqueName: \"kubernetes.io/projected/51e28897-c906-4487-b6c5-c5d588d450c5-kube-api-access-8xlqs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:52.197642 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.197640 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e28897-c906-4487-b6c5-c5d588d450c5-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:58:52.680682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.680646 5946 generic.go:356] "Generic (PLEG): container finished" podID="51e28897-c906-4487-b6c5-c5d588d450c5" containerID="b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02" exitCode=0 Nov 03 16:58:52.680903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.680698 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" Nov 03 16:58:52.680903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.680741 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" event={"ID":"51e28897-c906-4487-b6c5-c5d588d450c5","Type":"ContainerDied","Data":"b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02"} Nov 03 16:58:52.680903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.680804 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp" event={"ID":"51e28897-c906-4487-b6c5-c5d588d450c5","Type":"ContainerDied","Data":"8dc3d3accd40ece45bf29f1496fd9f628b25ced22f9618dcdf8d1d8faf6065f7"} Nov 03 16:58:52.680903 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.680833 5946 scope.go:122] "RemoveContainer" containerID="b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02" Nov 03 16:58:52.691093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.691075 5946 scope.go:122] "RemoveContainer" containerID="b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02" Nov 03 16:58:52.691402 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:58:52.691380 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02\": container with ID starting with b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02 not found: ID does not exist" containerID="b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02" Nov 03 16:58:52.691479 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.691410 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02"} err="failed to get container status \"b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02\": rpc error: code = NotFound desc = could not find container \"b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02\": container with ID starting with b7ca60c7f274d1b95b859f33a29402850eac6c9ac2d2984ee9433e0623e85b02 not found: ID does not exist" Nov 03 16:58:52.852978 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.852941 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp"] Nov 03 16:58:52.903983 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:52.903952 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-244c65-85ccbffd75-fm8sp"] Nov 03 16:58:54.003854 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:54.003822 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e28897-c906-4487-b6c5-c5d588d450c5" path="/var/lib/kubelet/pods/51e28897-c906-4487-b6c5-c5d588d450c5/volumes" Nov 03 16:58:55.353968 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:55.353937 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:56.133251 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:56.133187 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" podUID="301b833f-e81c-4c4b-816c-669b7db045b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:58:56.842053 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:56.842023 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:57.753198 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.753163 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9"] Nov 03 16:58:57.753727 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.753712 5946 cpu_manager.go:414] "containerMap: removing container" podUID="51e28897-c906-4487-b6c5-c5d588d450c5" containerName="payload-processing" Nov 03 16:58:57.753781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.753728 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e28897-c906-4487-b6c5-c5d588d450c5" containerName="payload-processing" Nov 03 16:58:57.753781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.753739 5946 cpu_manager.go:414] "containerMap: removing container" podUID="358c779a-f372-4d84-948a-262e032e6dd9" containerName="revoke-keys" Nov 03 16:58:57.753781 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.753743 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358c779a-f372-4d84-948a-262e032e6dd9" containerName="revoke-keys" Nov 03 16:58:57.753877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.753828 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e28897-c906-4487-b6c5-c5d588d450c5" containerName="payload-processing" Nov 03 16:58:57.753877 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.753836 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="358c779a-f372-4d84-948a-262e032e6dd9" containerName="revoke-keys" Nov 03 16:58:57.756945 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.756927 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.816437 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.816403 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9"] Nov 03 16:58:57.862492 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.862464 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.862928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.862613 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9db\" (UniqueName: \"kubernetes.io/projected/794fb6a6-ff16-4e5c-8050-52642a53d11c-kube-api-access-7m9db\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.862928 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.862739 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.863688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.863651 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:58:57.863688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.863664 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:58:57.863848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.863715 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:58:57.863848 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.863729 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:58:57.964355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.964325 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.964541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.964384 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.964541 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.964476 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9db\" (UniqueName: \"kubernetes.io/projected/794fb6a6-ff16-4e5c-8050-52642a53d11c-kube-api-access-7m9db\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.965088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.965069 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:57.966773 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:57.966754 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:58.025132 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:58.025057 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9db\" (UniqueName: \"kubernetes.io/projected/794fb6a6-ff16-4e5c-8050-52642a53d11c-kube-api-access-7m9db\") pod \"maas-api-revoke-sub-6e1d4835-wrbw9\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:58.069805 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:58.069770 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:58:58.449128 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:58.449102 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9"] Nov 03 16:58:58.450823 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:58:58.450796 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794fb6a6_ff16_4e5c_8050_52642a53d11c.slice/crio-64b9f0b6c57b81c2ada791c6018725ee606294d416dd1925d8b9e0a4deac6e44 WatchSource:0}: Error finding container 64b9f0b6c57b81c2ada791c6018725ee606294d416dd1925d8b9e0a4deac6e44: Status 404 returned error can't find the container with id 64b9f0b6c57b81c2ada791c6018725ee606294d416dd1925d8b9e0a4deac6e44 Nov 03 16:58:58.720567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:58.720529 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" event={"ID":"794fb6a6-ff16-4e5c-8050-52642a53d11c","Type":"ContainerStarted","Data":"64b9f0b6c57b81c2ada791c6018725ee606294d416dd1925d8b9e0a4deac6e44"} Nov 03 16:58:59.728143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:59.728057 5946 generic.go:356] "Generic (PLEG): container finished" podID="794fb6a6-ff16-4e5c-8050-52642a53d11c" containerID="be5066902d6f998dc7c0525a1551e1598634e9d23dc8f14b54b7ca938a3ee81b" exitCode=0 Nov 03 16:58:59.728565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:58:59.728144 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" event={"ID":"794fb6a6-ff16-4e5c-8050-52642a53d11c","Type":"ContainerDied","Data":"be5066902d6f998dc7c0525a1551e1598634e9d23dc8f14b54b7ca938a3ee81b"} Nov 03 16:59:00.058861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:00.058795 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:00.885337 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:00.885312 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:59:00.998504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:00.998419 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-openshift-ca\") pod \"794fb6a6-ff16-4e5c-8050-52642a53d11c\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " Nov 03 16:59:00.998504 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:00.998464 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794fb6a6-ff16-4e5c-8050-52642a53d11c-kube-api-access-7m9db\" (UniqueName: \"kubernetes.io/projected/794fb6a6-ff16-4e5c-8050-52642a53d11c-kube-api-access-7m9db\") pod \"794fb6a6-ff16-4e5c-8050-52642a53d11c\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " Nov 03 16:59:00.998713 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:00.998542 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-xks-ca\") pod \"794fb6a6-ff16-4e5c-8050-52642a53d11c\" (UID: \"794fb6a6-ff16-4e5c-8050-52642a53d11c\") " Nov 03 16:59:00.998776 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:00.998756 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-openshift-ca" pod "794fb6a6-ff16-4e5c-8050-52642a53d11c" (UID: "794fb6a6-ff16-4e5c-8050-52642a53d11c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:00.998981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:00.998960 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:01.000898 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:01.000876 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794fb6a6-ff16-4e5c-8050-52642a53d11c-kube-api-access-7m9db" pod "794fb6a6-ff16-4e5c-8050-52642a53d11c" (UID: "794fb6a6-ff16-4e5c-8050-52642a53d11c"). InnerVolumeSpecName "kube-api-access-7m9db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:01.001027 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:01.000980 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-xks-ca" pod "794fb6a6-ff16-4e5c-8050-52642a53d11c" (UID: "794fb6a6-ff16-4e5c-8050-52642a53d11c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:01.100332 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:01.100301 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9db\" (UniqueName: \"kubernetes.io/projected/794fb6a6-ff16-4e5c-8050-52642a53d11c-kube-api-access-7m9db\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:01.100332 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:01.100329 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794fb6a6-ff16-4e5c-8050-52642a53d11c-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:01.742842 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:01.742811 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" Nov 03 16:59:01.743050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:01.742806 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9" event={"ID":"794fb6a6-ff16-4e5c-8050-52642a53d11c","Type":"ContainerDied","Data":"64b9f0b6c57b81c2ada791c6018725ee606294d416dd1925d8b9e0a4deac6e44"} Nov 03 16:59:01.743050 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:01.742959 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64b9f0b6c57b81c2ada791c6018725ee606294d416dd1925d8b9e0a4deac6e44" Nov 03 16:59:04.754939 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:04.754897 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:06.133552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:06.133507 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" podUID="301b833f-e81c-4c4b-816c-669b7db045b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:59:07.848147 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.848113 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww"] Nov 03 16:59:07.848689 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.848674 5946 cpu_manager.go:414] "containerMap: removing container" podUID="794fb6a6-ff16-4e5c-8050-52642a53d11c" containerName="revoke-keys" Nov 03 16:59:07.848753 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.848691 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794fb6a6-ff16-4e5c-8050-52642a53d11c" containerName="revoke-keys" Nov 03 16:59:07.848791 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.848780 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="794fb6a6-ff16-4e5c-8050-52642a53d11c" containerName="revoke-keys" Nov 03 16:59:07.852143 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.852125 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:07.912580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.912552 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww"] Nov 03 16:59:07.960005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.959968 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:59:07.960005 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.959986 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:59:07.960235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.960032 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:59:07.960235 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.960111 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:59:07.988926 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.988897 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:07.989095 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.989078 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhgv\" (UniqueName: \"kubernetes.io/projected/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-kube-api-access-rzhgv\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:07.989178 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:07.989158 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.090770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.090732 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhgv\" (UniqueName: \"kubernetes.io/projected/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-kube-api-access-rzhgv\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.090957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.090805 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.090957 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.090847 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.091490 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.091470 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.093390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.093371 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.151750 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.151681 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhgv\" (UniqueName: \"kubernetes.io/projected/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-kube-api-access-rzhgv\") pod \"maas-api-revoke-sub-96c285dc-m95ww\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.165551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.165521 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:08.551094 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.551064 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww"] Nov 03 16:59:08.553018 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:59:08.552986 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93917e2_6ca6_48b8_86ba_3d959b0bbba9.slice/crio-8582970425e4f4635526547cae53f8721618b181d8751fe294c0c6e7fdcee1b3 WatchSource:0}: Error finding container 8582970425e4f4635526547cae53f8721618b181d8751fe294c0c6e7fdcee1b3: Status 404 returned error can't find the container with id 8582970425e4f4635526547cae53f8721618b181d8751fe294c0c6e7fdcee1b3 Nov 03 16:59:08.790024 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:08.789989 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" event={"ID":"e93917e2-6ca6-48b8-86ba-3d959b0bbba9","Type":"ContainerStarted","Data":"8582970425e4f4635526547cae53f8721618b181d8751fe294c0c6e7fdcee1b3"} Nov 03 16:59:09.458837 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:09.458800 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:09.798396 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:09.798359 5946 generic.go:356] "Generic (PLEG): container finished" podID="e93917e2-6ca6-48b8-86ba-3d959b0bbba9" containerID="ba7e1f80cbb39d4be57419f7cc3d2b95a2c698f67da23926fc54c6b3e3a4c26e" exitCode=0 Nov 03 16:59:09.798597 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:09.798450 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" event={"ID":"e93917e2-6ca6-48b8-86ba-3d959b0bbba9","Type":"ContainerDied","Data":"ba7e1f80cbb39d4be57419f7cc3d2b95a2c698f67da23926fc54c6b3e3a4c26e"} Nov 03 16:59:10.949771 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:10.949742 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:11.124311 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.124229 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-kube-api-access-rzhgv\" (UniqueName: \"kubernetes.io/projected/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-kube-api-access-rzhgv\") pod \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " Nov 03 16:59:11.124470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.124364 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-xks-ca\") pod \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " Nov 03 16:59:11.124470 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.124418 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-openshift-ca\") pod \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\" (UID: \"e93917e2-6ca6-48b8-86ba-3d959b0bbba9\") " Nov 03 16:59:11.124863 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.124803 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-openshift-ca" pod "e93917e2-6ca6-48b8-86ba-3d959b0bbba9" (UID: "e93917e2-6ca6-48b8-86ba-3d959b0bbba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:11.126551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.126524 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-xks-ca" pod "e93917e2-6ca6-48b8-86ba-3d959b0bbba9" (UID: "e93917e2-6ca6-48b8-86ba-3d959b0bbba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:11.126551 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.126544 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-kube-api-access-rzhgv" pod "e93917e2-6ca6-48b8-86ba-3d959b0bbba9" (UID: "e93917e2-6ca6-48b8-86ba-3d959b0bbba9"). InnerVolumeSpecName "kube-api-access-rzhgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:11.225827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.225789 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzhgv\" (UniqueName: \"kubernetes.io/projected/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-kube-api-access-rzhgv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:11.225827 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.225824 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:11.226021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.225834 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93917e2-6ca6-48b8-86ba-3d959b0bbba9-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:11.811674 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.811639 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" Nov 03 16:59:11.811858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.811669 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww" event={"ID":"e93917e2-6ca6-48b8-86ba-3d959b0bbba9","Type":"ContainerDied","Data":"8582970425e4f4635526547cae53f8721618b181d8751fe294c0c6e7fdcee1b3"} Nov 03 16:59:11.811858 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:11.811721 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8582970425e4f4635526547cae53f8721618b181d8751fe294c0c6e7fdcee1b3" Nov 03 16:59:12.151944 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.151920 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:59:12.336751 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.336718 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/301b833f-e81c-4c4b-816c-669b7db045b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301b833f-e81c-4c4b-816c-669b7db045b1-plugins-config-volume\") pod \"301b833f-e81c-4c4b-816c-669b7db045b1\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " Nov 03 16:59:12.336923 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.336762 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301b833f-e81c-4c4b-816c-669b7db045b1-kube-api-access-xhns5\" (UniqueName: \"kubernetes.io/projected/301b833f-e81c-4c4b-816c-669b7db045b1-kube-api-access-xhns5\") pod \"301b833f-e81c-4c4b-816c-669b7db045b1\" (UID: \"301b833f-e81c-4c4b-816c-669b7db045b1\") " Nov 03 16:59:12.337172 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.337148 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301b833f-e81c-4c4b-816c-669b7db045b1-plugins-config-volume" pod "301b833f-e81c-4c4b-816c-669b7db045b1" (UID: "301b833f-e81c-4c4b-816c-669b7db045b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:12.339106 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.339087 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301b833f-e81c-4c4b-816c-669b7db045b1-kube-api-access-xhns5" pod "301b833f-e81c-4c4b-816c-669b7db045b1" (UID: "301b833f-e81c-4c4b-816c-669b7db045b1"). InnerVolumeSpecName "kube-api-access-xhns5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:12.437651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.437571 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301b833f-e81c-4c4b-816c-669b7db045b1-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:12.437651 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.437599 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhns5\" (UniqueName: \"kubernetes.io/projected/301b833f-e81c-4c4b-816c-669b7db045b1-kube-api-access-xhns5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:12.586022 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:59:12.585976 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:59:12.608982 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.608952 5946 scope.go:122] "RemoveContainer" containerID="48ec2041b5aa69d03783281a6822497f6821b5501ca457daedec759809330e0a" Nov 03 16:59:12.818180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.818140 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" event={"ID":"301b833f-e81c-4c4b-816c-669b7db045b1","Type":"ContainerDied","Data":"48ec2041b5aa69d03783281a6822497f6821b5501ca457daedec759809330e0a"} Nov 03 16:59:12.818180 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.818177 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" Nov 03 16:59:12.818414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:12.818191 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r" event={"ID":"301b833f-e81c-4c4b-816c-669b7db045b1","Type":"ContainerDied","Data":"eac05c8abe0935e7eda1539a157e7e5feb8f80ce49c8c7489a446dbd11faa5a6"} Nov 03 16:59:13.056036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:13.056000 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r"] Nov 03 16:59:13.062590 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:13.062559 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-244c65-7dd54dd87f-49b5r"] Nov 03 16:59:14.005486 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.005446 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="301b833f-e81c-4c4b-816c-669b7db045b1" path="/var/lib/kubelet/pods/301b833f-e81c-4c4b-816c-669b7db045b1/volumes" Nov 03 16:59:14.162169 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.162119 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:14.472288 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.466815 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:14.930392 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.930355 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v"] Nov 03 16:59:14.930896 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.930883 5946 cpu_manager.go:414] "containerMap: removing container" podUID="301b833f-e81c-4c4b-816c-669b7db045b1" containerName="payload-pre-processing" Nov 03 16:59:14.930953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.930897 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301b833f-e81c-4c4b-816c-669b7db045b1" containerName="payload-pre-processing" Nov 03 16:59:14.930953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.930906 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e93917e2-6ca6-48b8-86ba-3d959b0bbba9" containerName="revoke-keys" Nov 03 16:59:14.930953 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.930911 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93917e2-6ca6-48b8-86ba-3d959b0bbba9" containerName="revoke-keys" Nov 03 16:59:14.931064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.930992 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="301b833f-e81c-4c4b-816c-669b7db045b1" containerName="payload-pre-processing" Nov 03 16:59:14.931064 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.930999 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93917e2-6ca6-48b8-86ba-3d959b0bbba9" containerName="revoke-keys" Nov 03 16:59:14.935857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.935836 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:14.962930 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.962899 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:14.963056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.962938 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:14.963056 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:14.962962 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnqpn\" (UniqueName: \"kubernetes.io/projected/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-kube-api-access-cnqpn\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.010688 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.010647 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:59:15.011093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.010654 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:59:15.011093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.010692 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:59:15.011093 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.010712 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:59:15.013310 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.013289 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v"] Nov 03 16:59:15.064390 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.064349 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.064598 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.064437 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnqpn\" (UniqueName: \"kubernetes.io/projected/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-kube-api-access-cnqpn\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.064700 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.064682 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.065137 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.065112 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.067176 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.067158 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.125534 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.125504 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnqpn\" (UniqueName: \"kubernetes.io/projected/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-kube-api-access-cnqpn\") pod \"maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.249493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.249455 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:15.635849 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.635820 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v"] Nov 03 16:59:15.637695 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:59:15.637659 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90849e57_56f5_4ed4_b793_c8bf8ea54bc8.slice/crio-52160f0198f126fcee8a874b436626fcb22daaf2e360b31b7c3e488acbfadb2b WatchSource:0}: Error finding container 52160f0198f126fcee8a874b436626fcb22daaf2e360b31b7c3e488acbfadb2b: Status 404 returned error can't find the container with id 52160f0198f126fcee8a874b436626fcb22daaf2e360b31b7c3e488acbfadb2b Nov 03 16:59:15.839871 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:15.839837 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" event={"ID":"90849e57-56f5-4ed4-b793-c8bf8ea54bc8","Type":"ContainerStarted","Data":"52160f0198f126fcee8a874b436626fcb22daaf2e360b31b7c3e488acbfadb2b"} Nov 03 16:59:16.848433 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:16.848333 5946 generic.go:356] "Generic (PLEG): container finished" podID="90849e57-56f5-4ed4-b793-c8bf8ea54bc8" containerID="2c52acc2dbda653d453702b33c5041cb65cdfafbc95e0556386a9c432db843e9" exitCode=0 Nov 03 16:59:16.848861 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:16.848415 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" event={"ID":"90849e57-56f5-4ed4-b793-c8bf8ea54bc8","Type":"ContainerDied","Data":"2c52acc2dbda653d453702b33c5041cb65cdfafbc95e0556386a9c432db843e9"} Nov 03 16:59:18.020708 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.020676 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:18.112977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.112890 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-xks-ca\") pod \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " Nov 03 16:59:18.112977 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.112940 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-kube-api-access-cnqpn\" (UniqueName: \"kubernetes.io/projected/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-kube-api-access-cnqpn\") pod \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " Nov 03 16:59:18.113263 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.113051 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-openshift-ca\") pod \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\" (UID: \"90849e57-56f5-4ed4-b793-c8bf8ea54bc8\") " Nov 03 16:59:18.113423 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.113395 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-openshift-ca" pod "90849e57-56f5-4ed4-b793-c8bf8ea54bc8" (UID: "90849e57-56f5-4ed4-b793-c8bf8ea54bc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:18.113567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.113548 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:18.115373 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.115346 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-xks-ca" pod "90849e57-56f5-4ed4-b793-c8bf8ea54bc8" (UID: "90849e57-56f5-4ed4-b793-c8bf8ea54bc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:18.115472 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.115373 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-kube-api-access-cnqpn" pod "90849e57-56f5-4ed4-b793-c8bf8ea54bc8" (UID: "90849e57-56f5-4ed4-b793-c8bf8ea54bc8"). InnerVolumeSpecName "kube-api-access-cnqpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:18.215036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.215002 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:18.215036 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.215028 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnqpn\" (UniqueName: \"kubernetes.io/projected/90849e57-56f5-4ed4-b793-c8bf8ea54bc8-kube-api-access-cnqpn\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:18.861268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.861237 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" Nov 03 16:59:18.861464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.861235 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v" event={"ID":"90849e57-56f5-4ed4-b793-c8bf8ea54bc8","Type":"ContainerDied","Data":"52160f0198f126fcee8a874b436626fcb22daaf2e360b31b7c3e488acbfadb2b"} Nov 03 16:59:18.861464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:18.861362 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52160f0198f126fcee8a874b436626fcb22daaf2e360b31b7c3e488acbfadb2b" Nov 03 16:59:19.258105 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:19.258077 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:23.962355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:23.962309 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:26.740629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:26.740595 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:26.955189 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:26.955158 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:28.618643 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:28.618609 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 16:59:28.625399 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:28.625368 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:33.414170 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:33.414131 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:35.685775 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:35.685703 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:36.590948 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.590912 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2"] Nov 03 16:59:36.591465 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.591450 5946 cpu_manager.go:414] "containerMap: removing container" podUID="90849e57-56f5-4ed4-b793-c8bf8ea54bc8" containerName="revoke-keys" Nov 03 16:59:36.591520 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.591467 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90849e57-56f5-4ed4-b793-c8bf8ea54bc8" containerName="revoke-keys" Nov 03 16:59:36.591577 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.591567 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="90849e57-56f5-4ed4-b793-c8bf8ea54bc8" containerName="revoke-keys" Nov 03 16:59:36.594925 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.594903 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.656091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.656056 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:59:36.656091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.656071 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:59:36.656091 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.656080 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:59:36.709929 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.709896 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2"] Nov 03 16:59:36.728515 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.728485 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.728672 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.728650 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh96\" (UniqueName: \"kubernetes.io/projected/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-kube-api-access-phh96\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.728785 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.728766 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.758108 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.758084 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:59:36.829728 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.829697 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh96\" (UniqueName: \"kubernetes.io/projected/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-kube-api-access-phh96\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.829905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.829778 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.829905 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.829826 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.830447 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.830420 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.832511 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.832491 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.891231 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.891138 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh96\" (UniqueName: \"kubernetes.io/projected/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-kube-api-access-phh96\") pod \"maas-api-revoke-sub-3fd71af3-q86t2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:36.908255 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:36.908228 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:37.288887 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:37.288860 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2"] Nov 03 16:59:37.290806 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:59:37.290781 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4126a9b0_6167_4f41_87e6_676cc3dbb8b2.slice/crio-19d2a89215b92ce43735dece904904d3d90ab18076d5f7f35a26f5cdb65a6603 WatchSource:0}: Error finding container 19d2a89215b92ce43735dece904904d3d90ab18076d5f7f35a26f5cdb65a6603: Status 404 returned error can't find the container with id 19d2a89215b92ce43735dece904904d3d90ab18076d5f7f35a26f5cdb65a6603 Nov 03 16:59:37.579232 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:37.579123 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:37.990315 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:37.990279 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" event={"ID":"4126a9b0-6167-4f41-87e6-676cc3dbb8b2","Type":"ContainerStarted","Data":"64462b4d5c99299c5a74ff38757b970778d53741d48407c9f5628814533eaa08"} Nov 03 16:59:37.990719 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:37.990338 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" event={"ID":"4126a9b0-6167-4f41-87e6-676cc3dbb8b2","Type":"ContainerStarted","Data":"19d2a89215b92ce43735dece904904d3d90ab18076d5f7f35a26f5cdb65a6603"} Nov 03 16:59:38.116634 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:38.116568 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" podStartSLOduration=2.116546725 podStartE2EDuration="2.116546725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:38.113465444 +0000 UTC m=+2848.234518289" watchObservedRunningTime="2026-11-03 16:59:38.116546725 +0000 UTC m=+2848.237599551" Nov 03 16:59:38.998563 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:38.998531 5946 generic.go:356] "Generic (PLEG): container finished" podID="4126a9b0-6167-4f41-87e6-676cc3dbb8b2" containerID="64462b4d5c99299c5a74ff38757b970778d53741d48407c9f5628814533eaa08" exitCode=0 Nov 03 16:59:38.999007 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:38.998615 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" event={"ID":"4126a9b0-6167-4f41-87e6-676cc3dbb8b2","Type":"ContainerDied","Data":"64462b4d5c99299c5a74ff38757b970778d53741d48407c9f5628814533eaa08"} Nov 03 16:59:39.052233 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:39.052165 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:39.271879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:39.271784 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:39.461536 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:39.461506 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:39.548682 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:39.548601 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb"] Nov 03 16:59:39.553694 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:39.553668 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-244c65-lfwbb"] Nov 03 16:59:39.948525 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:39.948450 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:40.004667 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.004637 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41386308-3a42-4da4-88ca-0d2901d13872" path="/var/lib/kubelet/pods/41386308-3a42-4da4-88ca-0d2901d13872/volumes" Nov 03 16:59:40.156464 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.156435 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:40.278416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.278383 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-openshift-ca\") pod \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " Nov 03 16:59:40.278605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.278465 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-xks-ca\") pod \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " Nov 03 16:59:40.278605 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.278504 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-kube-api-access-phh96\" (UniqueName: \"kubernetes.io/projected/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-kube-api-access-phh96\") pod \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\" (UID: \"4126a9b0-6167-4f41-87e6-676cc3dbb8b2\") " Nov 03 16:59:40.279493 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.279447 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-openshift-ca" pod "4126a9b0-6167-4f41-87e6-676cc3dbb8b2" (UID: "4126a9b0-6167-4f41-87e6-676cc3dbb8b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:40.281621 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.281587 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-kube-api-access-phh96" pod "4126a9b0-6167-4f41-87e6-676cc3dbb8b2" (UID: "4126a9b0-6167-4f41-87e6-676cc3dbb8b2"). InnerVolumeSpecName "kube-api-access-phh96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:40.281917 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.281886 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-xks-ca" pod "4126a9b0-6167-4f41-87e6-676cc3dbb8b2" (UID: "4126a9b0-6167-4f41-87e6-676cc3dbb8b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:40.379907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.379869 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:40.379907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.379902 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:40.379907 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.379911 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh96\" (UniqueName: \"kubernetes.io/projected/4126a9b0-6167-4f41-87e6-676cc3dbb8b2-kube-api-access-phh96\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:40.517372 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:40.517329 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:41.012092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:41.012062 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" Nov 03 16:59:41.012092 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:41.012069 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2" event={"ID":"4126a9b0-6167-4f41-87e6-676cc3dbb8b2","Type":"ContainerDied","Data":"19d2a89215b92ce43735dece904904d3d90ab18076d5f7f35a26f5cdb65a6603"} Nov 03 16:59:41.012584 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:41.012114 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19d2a89215b92ce43735dece904904d3d90ab18076d5f7f35a26f5cdb65a6603" Nov 03 16:59:42.586261 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:59:42.586200 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 16:59:44.370784 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:44.370748 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:48.292044 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:48.292008 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:48.495999 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:48.495968 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:48.620386 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:48.620297 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw"] Nov 03 16:59:48.620575 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:48.620509 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" containerID="cri-o://3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0" gracePeriod=30 Nov 03 16:59:48.690774 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:48.690742 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:49.461565 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.461530 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2"] Nov 03 16:59:49.462021 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.462009 5946 cpu_manager.go:414] "containerMap: removing container" podUID="4126a9b0-6167-4f41-87e6-676cc3dbb8b2" containerName="revoke-keys" Nov 03 16:59:49.462079 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.462022 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4126a9b0-6167-4f41-87e6-676cc3dbb8b2" containerName="revoke-keys" Nov 03 16:59:49.462162 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.462153 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="4126a9b0-6167-4f41-87e6-676cc3dbb8b2" containerName="revoke-keys" Nov 03 16:59:49.465567 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.465548 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.498123 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.498091 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgljg\" (UniqueName: \"kubernetes.io/projected/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-kube-api-access-vgljg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.498295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.498144 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.498295 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.498269 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.525394 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.525356 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2"] Nov 03 16:59:49.573090 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.573057 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:59:49.573317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.573124 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:59:49.573317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.573173 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 16:59:49.573317 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.573288 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:59:49.599835 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.599805 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.600018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.599883 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.600018 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.600006 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgljg\" (UniqueName: \"kubernetes.io/projected/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-kube-api-access-vgljg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.600564 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.600540 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.602452 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.602432 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.660580 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.660549 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgljg\" (UniqueName: \"kubernetes.io/projected/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-kube-api-access-vgljg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:49.781075 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:49.781042 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:50.054484 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:50.054402 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:50.166080 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:50.166053 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2"] Nov 03 16:59:50.167549 ip-10-0-135-216 kubenswrapper[5946]: W1103 16:59:50.167516 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92cb43e_64e5_4b3d_b5c8_d16b7df2a7b4.slice/crio-3175f823c3e7510279569234c3b962ec056c2497533f002bbace1b0bca316ed0 WatchSource:0}: Error finding container 3175f823c3e7510279569234c3b962ec056c2497533f002bbace1b0bca316ed0: Status 404 returned error can't find the container with id 3175f823c3e7510279569234c3b962ec056c2497533f002bbace1b0bca316ed0 Nov 03 16:59:50.336967 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:50.336892 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:50.531134 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:50.531103 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:50.614387 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:50.614300 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v"] Nov 03 16:59:50.618979 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:50.618951 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7e55c3-vq58v"] Nov 03 16:59:51.074008 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:51.073975 5946 generic.go:356] "Generic (PLEG): container finished" podID="e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" containerID="57cea5304faa3b5d2639ce37ce23011af68579389fdd60544a377173a6787125" exitCode=0 Nov 03 16:59:51.074187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:51.074058 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" event={"ID":"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4","Type":"ContainerDied","Data":"57cea5304faa3b5d2639ce37ce23011af68579389fdd60544a377173a6787125"} Nov 03 16:59:51.074187 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:51.074109 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" event={"ID":"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4","Type":"ContainerStarted","Data":"3175f823c3e7510279569234c3b962ec056c2497533f002bbace1b0bca316ed0"} Nov 03 16:59:51.098660 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:51.098622 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:52.003954 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.003923 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90849e57-56f5-4ed4-b793-c8bf8ea54bc8" path="/var/lib/kubelet/pods/90849e57-56f5-4ed4-b793-c8bf8ea54bc8/volumes" Nov 03 16:59:52.227355 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.227323 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:52.331228 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.331138 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-xks-ca\") pod \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " Nov 03 16:59:52.331375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.331236 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-kube-api-access-vgljg\" (UniqueName: \"kubernetes.io/projected/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-kube-api-access-vgljg\") pod \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " Nov 03 16:59:52.331375 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.331307 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-openshift-ca\") pod \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\" (UID: \"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4\") " Nov 03 16:59:52.331623 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.331594 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-openshift-ca" pod "e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" (UID: "e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:52.331738 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.331720 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:52.333414 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.333393 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-xks-ca" pod "e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" (UID: "e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:52.333506 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.333430 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-kube-api-access-vgljg" pod "e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" (UID: "e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4"). InnerVolumeSpecName "kube-api-access-vgljg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:52.432552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.432518 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgljg\" (UniqueName: \"kubernetes.io/projected/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-kube-api-access-vgljg\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:52.432552 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.432548 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:52.987088 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.987043 5946 patch_prober.go:29] interesting pod/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:59:52.987275 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:52.987104 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:59:53.087305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.087274 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" event={"ID":"e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4","Type":"ContainerDied","Data":"3175f823c3e7510279569234c3b962ec056c2497533f002bbace1b0bca316ed0"} Nov 03 16:59:53.087305 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.087305 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2" Nov 03 16:59:53.087734 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.087321 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3175f823c3e7510279569234c3b962ec056c2497533f002bbace1b0bca316ed0" Nov 03 16:59:53.877416 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.877391 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:59:53.949840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.949768 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-socket\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.949840 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.949801 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-envoy\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.949849 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-token\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-token\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.949897 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33689f9-d3d1-4959-8112-4b7a2755a4af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33689f9-d3d1-4959-8112-4b7a2755a4af-istiod-ca-cert\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.949946 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-kube-api-access-qqj6w\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-kube-api-access-qqj6w\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950045 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950015 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-podinfo\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950064 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-credential-socket\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950090 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-certs\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950103 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-socket" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:53.950268 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950134 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-data\") pod \"d33689f9-d3d1-4959-8112-4b7a2755a4af\" (UID: \"d33689f9-d3d1-4959-8112-4b7a2755a4af\") " Nov 03 16:59:53.950476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950422 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-credential-socket" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:53.950476 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950433 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-certs" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:53.950609 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950579 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-data" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:53.950671 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950638 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33689f9-d3d1-4959-8112-4b7a2755a4af-istiod-ca-cert" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:53.950726 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950662 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:53.950770 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950698 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:53.950879 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.950836 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:53.952313 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.952289 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-token" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:53.952313 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.952299 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-envoy" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:53.952502 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.952416 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-podinfo" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:59:53.952740 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:53.952722 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-kube-api-access-qqj6w" pod "d33689f9-d3d1-4959-8112-4b7a2755a4af" (UID: "d33689f9-d3d1-4959-8112-4b7a2755a4af"). InnerVolumeSpecName "kube-api-access-qqj6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:54.052421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.052382 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqj6w\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-kube-api-access-qqj6w\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:54.052421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.052415 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:54.052421 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.052428 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:54.052741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.052441 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:54.052741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.052453 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33689f9-d3d1-4959-8112-4b7a2755a4af-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:54.052741 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.052464 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33689f9-d3d1-4959-8112-4b7a2755a4af-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 16:59:54.095395 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.095363 5946 generic.go:356] "Generic (PLEG): container finished" podID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerID="3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0" exitCode=0 Nov 03 16:59:54.095880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.095416 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" Nov 03 16:59:54.095880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.095453 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" event={"ID":"d33689f9-d3d1-4959-8112-4b7a2755a4af","Type":"ContainerDied","Data":"3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0"} Nov 03 16:59:54.095880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.095522 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw" event={"ID":"d33689f9-d3d1-4959-8112-4b7a2755a4af","Type":"ContainerDied","Data":"3b2ad3f815af378b61e875d5d3b4705c1c50d573bdd9f99d93f8715198fca7fe"} Nov 03 16:59:54.095880 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.095558 5946 scope.go:122] "RemoveContainer" containerID="3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0" Nov 03 16:59:54.105612 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.105590 5946 scope.go:122] "RemoveContainer" containerID="cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a" Nov 03 16:59:54.118291 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.117910 5946 scope.go:122] "RemoveContainer" containerID="3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0" Nov 03 16:59:54.118717 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:59:54.118566 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0\": container with ID starting with 3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0 not found: ID does not exist" containerID="3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0" Nov 03 16:59:54.118717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.118607 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0"} err="failed to get container status \"3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0\": rpc error: code = NotFound desc = could not find container \"3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0\": container with ID starting with 3ca3c5029afe890daaadd3fa7e54c21df9475e9f9798fef2e202a416501d72f0 not found: ID does not exist" Nov 03 16:59:54.118717 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.118636 5946 scope.go:122] "RemoveContainer" containerID="cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a" Nov 03 16:59:54.119464 ip-10-0-135-216 kubenswrapper[5946]: E1103 16:59:54.119441 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a\": container with ID starting with cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a not found: ID does not exist" containerID="cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a" Nov 03 16:59:54.119596 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.119471 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a"} err="failed to get container status \"cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a\": rpc error: code = NotFound desc = could not find container \"cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a\": container with ID starting with cd2514b687f3c4c92469bdbd3ad84d765e1a48badbba1e00bc9c1693cf24fe2a not found: ID does not exist" Nov 03 16:59:54.268586 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.268554 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw"] Nov 03 16:59:54.274981 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:54.274952 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-244c65-openshift-default-86f8d6646f-rpxbw"] Nov 03 16:59:55.467910 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:55.467868 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:56.003956 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:56.003915 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" path="/var/lib/kubelet/pods/d33689f9-d3d1-4959-8112-4b7a2755a4af/volumes" Nov 03 16:59:58.759705 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:58.759673 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:58.952857 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:58.952820 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:59.101629 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:59.101550 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx"] Nov 03 16:59:59.101779 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:59.101741 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" containerID="cri-o://3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580" gracePeriod=30 Nov 03 16:59:59.204870 ip-10-0-135-216 kubenswrapper[5946]: I1103 16:59:59.204799 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:00.115142 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115103 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln"] Nov 03 17:00:00.115636 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115620 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" Nov 03 17:00:00.115686 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115637 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" Nov 03 17:00:00.115686 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115665 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" Nov 03 17:00:00.115686 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115670 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" Nov 03 17:00:00.115686 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115679 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" containerName="revoke-keys" Nov 03 17:00:00.115686 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115685 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" containerName="revoke-keys" Nov 03 17:00:00.115846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115775 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" Nov 03 17:00:00.115846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.115784 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" containerName="revoke-keys" Nov 03 17:00:00.120766 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.120745 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.172819 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.172770 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s"] Nov 03 17:00:00.173574 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.173553 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33689f9-d3d1-4959-8112-4b7a2755a4af" containerName="istio-proxy" Nov 03 17:00:00.176908 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.176884 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.179673 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.179650 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm"] Nov 03 17:00:00.183454 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.183435 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb"] Nov 03 17:00:00.183558 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.183545 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.187301 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.187283 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.217582 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.217541 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.217769 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.217587 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7x5\" (UniqueName: \"kubernetes.io/projected/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-kube-api-access-qh7x5\") pod \"maas-api-key-cleanup-29895420-n9f6s\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.217769 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.217671 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895420-n9f6s\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.217769 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.217740 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486gq\" (UniqueName: \"kubernetes.io/projected/ce64b281-c110-441c-90e5-eff0960988b1-kube-api-access-486gq\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.217902 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.217785 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.217902 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.217812 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m55l\" (UniqueName: \"kubernetes.io/projected/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-kube-api-access-6m55l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.217994 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.217893 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19311f79-4585-4ddf-894c-a3ba0577d9bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.218062 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.218035 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72tk\" (UniqueName: \"kubernetes.io/projected/19311f79-4585-4ddf-894c-a3ba0577d9bf-kube-api-access-x72tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.218146 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.218124 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.228255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.220963 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:00:00.228255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.221793 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:00:00.228255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.225354 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f"] Nov 03 17:00:00.228255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.227688 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:00:00.228255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.227797 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:00:00.230516 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.230485 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln"] Nov 03 17:00:00.230654 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.230608 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.233626 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.233599 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn"] Nov 03 17:00:00.238376 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.238354 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln"] Nov 03 17:00:00.238520 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.238469 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.243162 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.243137 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.264277 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.264249 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s"] Nov 03 17:00:00.270410 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.270383 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:00.279891 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.279651 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm"] Nov 03 17:00:00.287794 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.287766 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f"] Nov 03 17:00:00.319231 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319177 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kh5\" (UniqueName: \"kubernetes.io/projected/42492a4a-b6b9-4f1e-91d1-dd508fa57852-kube-api-access-p9kh5\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.319456 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319310 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d396844f-cfd1-485c-a52f-33f6d944acd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.319456 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319377 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.319456 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319425 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7x5\" (UniqueName: \"kubernetes.io/projected/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-kube-api-access-qh7x5\") pod \"maas-api-key-cleanup-29895420-n9f6s\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.319456 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319453 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42492a4a-b6b9-4f1e-91d1-dd508fa57852-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.319786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319492 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895420-n9f6s\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.319786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319522 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpcgr\" (UniqueName: \"kubernetes.io/projected/d396844f-cfd1-485c-a52f-33f6d944acd1-kube-api-access-cpcgr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.319786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319554 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486gq\" (UniqueName: \"kubernetes.io/projected/ce64b281-c110-441c-90e5-eff0960988b1-kube-api-access-486gq\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.319786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319636 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.319786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319666 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx68p\" (UniqueName: \"kubernetes.io/projected/7812793b-f556-46a7-8993-cfe7fa411179-kube-api-access-jx68p\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.319786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319707 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m55l\" (UniqueName: \"kubernetes.io/projected/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-kube-api-access-6m55l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.319786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319778 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19311f79-4585-4ddf-894c-a3ba0577d9bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.320219 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319811 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72tk\" (UniqueName: \"kubernetes.io/projected/19311f79-4585-4ddf-894c-a3ba0577d9bf-kube-api-access-x72tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.320219 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319863 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812793b-f556-46a7-8993-cfe7fa411179-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.320219 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.319921 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.320930 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.320902 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.321648 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.321622 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.322675 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.322651 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895420-n9f6s\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.324064 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.323957 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19311f79-4585-4ddf-894c-a3ba0577d9bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.326060 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.326038 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn"] Nov 03 17:00:00.341083 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.327836 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.341083 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.330119 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln"] Nov 03 17:00:00.341083 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.334574 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb"] Nov 03 17:00:00.382898 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.382814 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7x5\" (UniqueName: \"kubernetes.io/projected/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-kube-api-access-qh7x5\") pod \"maas-api-key-cleanup-29895420-n9f6s\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.385294 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.385268 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-486gq\" (UniqueName: \"kubernetes.io/projected/ce64b281-c110-441c-90e5-eff0960988b1-kube-api-access-486gq\") pod \"maas-api-revoke-keys-e2e-inf-b-61be78-q4fln\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.385418 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.385361 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72tk\" (UniqueName: \"kubernetes.io/projected/19311f79-4585-4ddf-894c-a3ba0577d9bf-kube-api-access-x72tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.388698 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.388675 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m55l\" (UniqueName: \"kubernetes.io/projected/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-kube-api-access-6m55l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.421097 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421070 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812793b-f556-46a7-8993-cfe7fa411179-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.421303 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421130 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kh5\" (UniqueName: \"kubernetes.io/projected/42492a4a-b6b9-4f1e-91d1-dd508fa57852-kube-api-access-p9kh5\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.421303 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421251 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d396844f-cfd1-485c-a52f-33f6d944acd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.421449 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421347 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42492a4a-b6b9-4f1e-91d1-dd508fa57852-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.421449 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421406 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpcgr\" (UniqueName: \"kubernetes.io/projected/d396844f-cfd1-485c-a52f-33f6d944acd1-kube-api-access-cpcgr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.421565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421515 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx68p\" (UniqueName: \"kubernetes.io/projected/7812793b-f556-46a7-8993-cfe7fa411179-kube-api-access-jx68p\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.421844 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421825 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812793b-f556-46a7-8993-cfe7fa411179-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.422006 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.421984 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d396844f-cfd1-485c-a52f-33f6d944acd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.422048 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.422012 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42492a4a-b6b9-4f1e-91d1-dd508fa57852-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.436034 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.436012 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:00.483524 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.483489 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpcgr\" (UniqueName: \"kubernetes.io/projected/d396844f-cfd1-485c-a52f-33f6d944acd1-kube-api-access-cpcgr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.484051 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.484032 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kh5\" (UniqueName: \"kubernetes.io/projected/42492a4a-b6b9-4f1e-91d1-dd508fa57852-kube-api-access-p9kh5\") pod \"maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.485348 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.485312 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx68p\" (UniqueName: \"kubernetes.io/projected/7812793b-f556-46a7-8993-cfe7fa411179-kube-api-access-jx68p\") pod \"maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.492528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.492502 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:00.500436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.500416 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:00.510133 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.508013 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:00.566588 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.558422 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:00.567236 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.567147 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:00.578020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.577908 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:00.824197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.824173 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln"] Nov 03 17:00:00.826711 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:00:00.826670 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce64b281_c110_441c_90e5_eff0960988b1.slice/crio-b658498f1add5aae410501243bac09271aa8e99f8f5e43e8a7a296799f326d0d WatchSource:0}: Error finding container b658498f1add5aae410501243bac09271aa8e99f8f5e43e8a7a296799f326d0d: Status 404 returned error can't find the container with id b658498f1add5aae410501243bac09271aa8e99f8f5e43e8a7a296799f326d0d Nov 03 17:00:00.911122 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.911096 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s"] Nov 03 17:00:00.913198 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:00:00.913157 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bdb396_45dd_42ab_be4b_fb8cc7f946d4.slice/crio-297b998edad50a576dfc4146cc747335f1be7c09812f4a5775ba56c2cad70b70 WatchSource:0}: Error finding container 297b998edad50a576dfc4146cc747335f1be7c09812f4a5775ba56c2cad70b70: Status 404 returned error can't find the container with id 297b998edad50a576dfc4146cc747335f1be7c09812f4a5775ba56c2cad70b70 Nov 03 17:00:00.922860 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:00.922837 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb"] Nov 03 17:00:00.925501 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:00:00.925471 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc28cee_8a22_4f6d_9f6a_48181d54e3ba.slice/crio-ec061283f7b8f9e9ff1f429b39bba50f541620adbf296530972585804b0eb84a WatchSource:0}: Error finding container ec061283f7b8f9e9ff1f429b39bba50f541620adbf296530972585804b0eb84a: Status 404 returned error can't find the container with id ec061283f7b8f9e9ff1f429b39bba50f541620adbf296530972585804b0eb84a Nov 03 17:00:01.003364 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.003335 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f"] Nov 03 17:00:01.008650 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.008607 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn"] Nov 03 17:00:01.012167 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.012147 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln"] Nov 03 17:00:01.012301 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:00:01.012278 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7812793b_f556_46a7_8993_cfe7fa411179.slice/crio-0381854cdce5a32099644bf3d49bfc9f8e0fe99bfd958d344ae9fed5c5573840 WatchSource:0}: Error finding container 0381854cdce5a32099644bf3d49bfc9f8e0fe99bfd958d344ae9fed5c5573840: Status 404 returned error can't find the container with id 0381854cdce5a32099644bf3d49bfc9f8e0fe99bfd958d344ae9fed5c5573840 Nov 03 17:00:01.032516 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.032493 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm"] Nov 03 17:00:01.118001 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:00:01.117964 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19311f79_4585_4ddf_894c_a3ba0577d9bf.slice/crio-efac5f7e34b8ceab11b4664f2d4323a3d1bcf97db4bf5d4cbd352c68d8094389 WatchSource:0}: Error finding container efac5f7e34b8ceab11b4664f2d4323a3d1bcf97db4bf5d4cbd352c68d8094389: Status 404 returned error can't find the container with id efac5f7e34b8ceab11b4664f2d4323a3d1bcf97db4bf5d4cbd352c68d8094389 Nov 03 17:00:01.146434 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.146402 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerStarted","Data":"0381854cdce5a32099644bf3d49bfc9f8e0fe99bfd958d344ae9fed5c5573840"} Nov 03 17:00:01.148270 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.148196 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerStarted","Data":"efac5f7e34b8ceab11b4664f2d4323a3d1bcf97db4bf5d4cbd352c68d8094389"} Nov 03 17:00:01.152689 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.152644 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" event={"ID":"ce64b281-c110-441c-90e5-eff0960988b1","Type":"ContainerStarted","Data":"b658498f1add5aae410501243bac09271aa8e99f8f5e43e8a7a296799f326d0d"} Nov 03 17:00:01.154328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.154305 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" event={"ID":"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba","Type":"ContainerStarted","Data":"ec061283f7b8f9e9ff1f429b39bba50f541620adbf296530972585804b0eb84a"} Nov 03 17:00:01.155537 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.155513 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" event={"ID":"32bdb396-45dd-42ab-be4b-fb8cc7f946d4","Type":"ContainerStarted","Data":"297b998edad50a576dfc4146cc747335f1be7c09812f4a5775ba56c2cad70b70"} Nov 03 17:00:01.156833 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.156775 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerStarted","Data":"c3dbf3a780f5c8ebec4307ab07b5597b640b94bfe9d796ab77160944ac00fa4f"} Nov 03 17:00:01.158006 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:01.157979 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerStarted","Data":"21114e42ac17f59dbbfb3556be42ebbfafad5e6819ac7a143270d8d02a56cb14"} Nov 03 17:00:02.166547 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.166442 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerStarted","Data":"96476b5735249835facff6334da45afcd1dcc1da90cc0fd148b4876a2d5f47c2"} Nov 03 17:00:02.168429 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.168378 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerStarted","Data":"0d36c1308f697238c1f0fbda9259563c812b251d98569e5ffac1629ae5f4e990"} Nov 03 17:00:02.170329 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.170299 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerStarted","Data":"7e0258e039dde4bf9a71cd5ad21c7ddee2918f76ad1d5f93760bb449fb2f6e32"} Nov 03 17:00:02.172139 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.172116 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerStarted","Data":"b42e29b5d353925972c74cc3a69a1151c0406afe4a002e9a69d030722883a8ea"} Nov 03 17:00:02.174006 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.173982 5946 generic.go:356] "Generic (PLEG): container finished" podID="ce64b281-c110-441c-90e5-eff0960988b1" containerID="bb72dccdfd607bc43764d457e6c8fe3709f50d4ddcde4a5dc4522b7a5271fbfb" exitCode=0 Nov 03 17:00:02.174119 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.174055 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" event={"ID":"ce64b281-c110-441c-90e5-eff0960988b1","Type":"ContainerDied","Data":"bb72dccdfd607bc43764d457e6c8fe3709f50d4ddcde4a5dc4522b7a5271fbfb"} Nov 03 17:00:02.176942 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.176916 5946 generic.go:356] "Generic (PLEG): container finished" podID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerID="9737e920fe9829922954ad3760c7db8d9342ed2803d0490f4cee6de85b113d29" exitCode=60 Nov 03 17:00:02.177472 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.177452 5946 scope.go:122] "RemoveContainer" containerID="9737e920fe9829922954ad3760c7db8d9342ed2803d0490f4cee6de85b113d29" Nov 03 17:00:02.177999 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.177973 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" event={"ID":"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba","Type":"ContainerDied","Data":"9737e920fe9829922954ad3760c7db8d9342ed2803d0490f4cee6de85b113d29"} Nov 03 17:00:02.180685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.180662 5946 generic.go:356] "Generic (PLEG): container finished" podID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerID="488ee2c9c1731e0b0beb5984b219d3f1c4949b159f031a9113fe9eedc0d2ca3a" exitCode=60 Nov 03 17:00:02.180771 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.180730 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" event={"ID":"32bdb396-45dd-42ab-be4b-fb8cc7f946d4","Type":"ContainerDied","Data":"488ee2c9c1731e0b0beb5984b219d3f1c4949b159f031a9113fe9eedc0d2ca3a"} Nov 03 17:00:02.180998 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.180977 5946 scope.go:122] "RemoveContainer" containerID="488ee2c9c1731e0b0beb5984b219d3f1c4949b159f031a9113fe9eedc0d2ca3a" Nov 03 17:00:02.287301 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.287241 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" podStartSLOduration=2.287226536 podStartE2EDuration="2.287226536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:02.285802172 +0000 UTC m=+2872.406854996" watchObservedRunningTime="2026-11-03 17:00:02.287226536 +0000 UTC m=+2872.408279416" Nov 03 17:00:02.458014 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.457978 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:02.524079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.524024 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" podStartSLOduration=2.524009688 podStartE2EDuration="2.524009688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:02.522171531 +0000 UTC m=+2872.643224355" watchObservedRunningTime="2026-11-03 17:00:02.524009688 +0000 UTC m=+2872.645062510" Nov 03 17:00:02.660445 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.660410 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:02.897040 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:02.896974 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" podStartSLOduration=2.896956201 podStartE2EDuration="2.896956201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:02.895708948 +0000 UTC m=+2873.016761795" watchObservedRunningTime="2026-11-03 17:00:02.896956201 +0000 UTC m=+2873.018009028" Nov 03 17:00:03.017976 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.017907 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" podStartSLOduration=3.017883721 podStartE2EDuration="3.017883721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:03.013905238 +0000 UTC m=+2873.134958053" watchObservedRunningTime="2026-11-03 17:00:03.017883721 +0000 UTC m=+2873.138936540" Nov 03 17:00:03.193159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.193113 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" event={"ID":"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba","Type":"ContainerStarted","Data":"f40e1064f3c84404f5f8c66fc76a95b416f8552c51a2409511cfae8bda4b56be"} Nov 03 17:00:03.195140 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.195111 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" event={"ID":"32bdb396-45dd-42ab-be4b-fb8cc7f946d4","Type":"ContainerStarted","Data":"9902906489fdf3c1e6d6aa61aba6a78f69bd71a99867c90ac82575117f169472"} Nov 03 17:00:03.196826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.196797 5946 generic.go:356] "Generic (PLEG): container finished" podID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerID="96476b5735249835facff6334da45afcd1dcc1da90cc0fd148b4876a2d5f47c2" exitCode=60 Nov 03 17:00:03.196963 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.196851 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerDied","Data":"96476b5735249835facff6334da45afcd1dcc1da90cc0fd148b4876a2d5f47c2"} Nov 03 17:00:03.197136 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.197121 5946 scope.go:122] "RemoveContainer" containerID="96476b5735249835facff6334da45afcd1dcc1da90cc0fd148b4876a2d5f47c2" Nov 03 17:00:03.199596 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.198933 5946 generic.go:356] "Generic (PLEG): container finished" podID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerID="0d36c1308f697238c1f0fbda9259563c812b251d98569e5ffac1629ae5f4e990" exitCode=60 Nov 03 17:00:03.199596 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.199020 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerDied","Data":"0d36c1308f697238c1f0fbda9259563c812b251d98569e5ffac1629ae5f4e990"} Nov 03 17:00:03.199596 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.199238 5946 scope.go:122] "RemoveContainer" containerID="0d36c1308f697238c1f0fbda9259563c812b251d98569e5ffac1629ae5f4e990" Nov 03 17:00:03.201063 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.201038 5946 generic.go:356] "Generic (PLEG): container finished" podID="7812793b-f556-46a7-8993-cfe7fa411179" containerID="7e0258e039dde4bf9a71cd5ad21c7ddee2918f76ad1d5f93760bb449fb2f6e32" exitCode=60 Nov 03 17:00:03.201191 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.201109 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerDied","Data":"7e0258e039dde4bf9a71cd5ad21c7ddee2918f76ad1d5f93760bb449fb2f6e32"} Nov 03 17:00:03.201331 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.201317 5946 scope.go:122] "RemoveContainer" containerID="7e0258e039dde4bf9a71cd5ad21c7ddee2918f76ad1d5f93760bb449fb2f6e32" Nov 03 17:00:03.202842 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.202823 5946 generic.go:356] "Generic (PLEG): container finished" podID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerID="b42e29b5d353925972c74cc3a69a1151c0406afe4a002e9a69d030722883a8ea" exitCode=60 Nov 03 17:00:03.202952 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.202883 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerDied","Data":"b42e29b5d353925972c74cc3a69a1151c0406afe4a002e9a69d030722883a8ea"} Nov 03 17:00:03.203058 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.203045 5946 scope.go:122] "RemoveContainer" containerID="b42e29b5d353925972c74cc3a69a1151c0406afe4a002e9a69d030722883a8ea" Nov 03 17:00:03.317607 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.314538 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" podStartSLOduration=3.314519006 podStartE2EDuration="3.314519006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:03.312897595 +0000 UTC m=+2873.433950418" watchObservedRunningTime="2026-11-03 17:00:03.314519006 +0000 UTC m=+2873.435571830" Nov 03 17:00:03.425539 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.425512 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:03.455580 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.455547 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce64b281-c110-441c-90e5-eff0960988b1-kube-api-access-486gq\" (UniqueName: \"kubernetes.io/projected/ce64b281-c110-441c-90e5-eff0960988b1-kube-api-access-486gq\") pod \"ce64b281-c110-441c-90e5-eff0960988b1\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " Nov 03 17:00:03.455769 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.455678 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-openshift-ca\") pod \"ce64b281-c110-441c-90e5-eff0960988b1\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " Nov 03 17:00:03.455818 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.455802 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-xks-ca\") pod \"ce64b281-c110-441c-90e5-eff0960988b1\" (UID: \"ce64b281-c110-441c-90e5-eff0960988b1\") " Nov 03 17:00:03.456041 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.456018 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-openshift-ca" pod "ce64b281-c110-441c-90e5-eff0960988b1" (UID: "ce64b281-c110-441c-90e5-eff0960988b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:03.456106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.456081 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:03.458019 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.457995 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-xks-ca" pod "ce64b281-c110-441c-90e5-eff0960988b1" (UID: "ce64b281-c110-441c-90e5-eff0960988b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:03.458098 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.458049 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce64b281-c110-441c-90e5-eff0960988b1-kube-api-access-486gq" pod "ce64b281-c110-441c-90e5-eff0960988b1" (UID: "ce64b281-c110-441c-90e5-eff0960988b1"). InnerVolumeSpecName "kube-api-access-486gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:03.552966 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.552915 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" podStartSLOduration=3.552902654 podStartE2EDuration="3.552902654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:03.551711167 +0000 UTC m=+2873.672764011" watchObservedRunningTime="2026-11-03 17:00:03.552902654 +0000 UTC m=+2873.673955477" Nov 03 17:00:03.557898 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.557867 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce64b281-c110-441c-90e5-eff0960988b1-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:03.558057 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:03.557903 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-486gq\" (UniqueName: \"kubernetes.io/projected/ce64b281-c110-441c-90e5-eff0960988b1-kube-api-access-486gq\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.223887 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.223585 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" event={"ID":"ce64b281-c110-441c-90e5-eff0960988b1","Type":"ContainerDied","Data":"b658498f1add5aae410501243bac09271aa8e99f8f5e43e8a7a296799f326d0d"} Nov 03 17:00:04.223887 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.223609 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln" Nov 03 17:00:04.223887 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.223689 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b658498f1add5aae410501243bac09271aa8e99f8f5e43e8a7a296799f326d0d" Nov 03 17:00:04.225899 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.225869 5946 generic.go:356] "Generic (PLEG): container finished" podID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerID="f40e1064f3c84404f5f8c66fc76a95b416f8552c51a2409511cfae8bda4b56be" exitCode=60 Nov 03 17:00:04.226016 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.225955 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" event={"ID":"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba","Type":"ContainerDied","Data":"f40e1064f3c84404f5f8c66fc76a95b416f8552c51a2409511cfae8bda4b56be"} Nov 03 17:00:04.226090 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.226077 5946 scope.go:122] "RemoveContainer" containerID="9737e920fe9829922954ad3760c7db8d9342ed2803d0490f4cee6de85b113d29" Nov 03 17:00:04.226304 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.226283 5946 scope.go:122] "RemoveContainer" containerID="f40e1064f3c84404f5f8c66fc76a95b416f8552c51a2409511cfae8bda4b56be" Nov 03 17:00:04.226738 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:04.226704 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb_odh-ai-gateway-infra(1dc28cee-8a22-4f6d-9f6a-48181d54e3ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" Nov 03 17:00:04.228641 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.228611 5946 generic.go:356] "Generic (PLEG): container finished" podID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerID="9902906489fdf3c1e6d6aa61aba6a78f69bd71a99867c90ac82575117f169472" exitCode=60 Nov 03 17:00:04.228743 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.228644 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" event={"ID":"32bdb396-45dd-42ab-be4b-fb8cc7f946d4","Type":"ContainerDied","Data":"9902906489fdf3c1e6d6aa61aba6a78f69bd71a99867c90ac82575117f169472"} Nov 03 17:00:04.228981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.228963 5946 scope.go:122] "RemoveContainer" containerID="9902906489fdf3c1e6d6aa61aba6a78f69bd71a99867c90ac82575117f169472" Nov 03 17:00:04.229342 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:04.229301 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895420-n9f6s_odh-ai-gateway-infra(32bdb396-45dd-42ab-be4b-fb8cc7f946d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" Nov 03 17:00:04.230967 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.230948 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerStarted","Data":"2bfe8eeed626d5a04fae0b400fad96f34871ceab238a7042d37dd559dfe92e72"} Nov 03 17:00:04.233726 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.233697 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerStarted","Data":"2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775"} Nov 03 17:00:04.233940 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.233922 5946 scope.go:122] "RemoveContainer" containerID="2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775" Nov 03 17:00:04.234266 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:04.234201 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln_odh-ai-gateway-infra(d396844f-cfd1-485c-a52f-33f6d944acd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" Nov 03 17:00:04.236356 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.236332 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerStarted","Data":"b51acf8bb7989f028d6cb2222b1b3a62529767ad81034b2729335ba0e42a212f"} Nov 03 17:00:04.238576 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.238552 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerStarted","Data":"b42de7c053d946fec107286630ff6cf5255040004852f11fd8d4558b15e24e1c"} Nov 03 17:00:04.297733 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.297699 5946 scope.go:122] "RemoveContainer" containerID="488ee2c9c1731e0b0beb5984b219d3f1c4949b159f031a9113fe9eedc0d2ca3a" Nov 03 17:00:04.394235 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.394175 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 17:00:04.468911 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.468880 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-kube-api-access-4zxx7\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-kube-api-access-4zxx7\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.469068 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.468941 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-socket\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.469068 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.469011 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-envoy\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.469363 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.469340 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-socket" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:04.469469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.469441 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-credential-socket" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:04.469624 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.469183 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-credential-socket\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.469778 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.469685 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-istio-token\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-istio-token\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.470469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.470069 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-data\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.474334 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474310 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-certs\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.474446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474357 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f797658-76a2-4569-b387-49b0680f7975-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f797658-76a2-4569-b387-49b0680f7975-istiod-ca-cert\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.474446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.470368 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-data" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:04.474446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.471805 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-kube-api-access-4zxx7" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "kube-api-access-4zxx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:04.474446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.471939 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-envoy" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:04.474446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474427 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f797658-76a2-4569-b387-49b0680f7975-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f797658-76a2-4569-b387-49b0680f7975-istio-podinfo\") pod \"6f797658-76a2-4569-b387-49b0680f7975\" (UID: \"6f797658-76a2-4569-b387-49b0680f7975\") " Nov 03 17:00:04.474715 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.472068 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-istio-token" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:04.474715 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474509 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-certs" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:04.474830 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474737 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f797658-76a2-4569-b387-49b0680f7975-istiod-ca-cert" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:04.474907 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474888 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zxx7\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-kube-api-access-4zxx7\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.474964 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474913 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.474964 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474927 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.474964 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474954 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.475118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474967 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f797658-76a2-4569-b387-49b0680f7975-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.475118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474981 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.475118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.474994 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f797658-76a2-4569-b387-49b0680f7975-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.475118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.475008 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f797658-76a2-4569-b387-49b0680f7975-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:04.476805 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.476780 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f797658-76a2-4569-b387-49b0680f7975-istio-podinfo" pod "6f797658-76a2-4569-b387-49b0680f7975" (UID: "6f797658-76a2-4569-b387-49b0680f7975"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:04.576775 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:04.576684 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f797658-76a2-4569-b387-49b0680f7975-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:05.247445 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.247408 5946 generic.go:356] "Generic (PLEG): container finished" podID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerID="b42de7c053d946fec107286630ff6cf5255040004852f11fd8d4558b15e24e1c" exitCode=60 Nov 03 17:00:05.247977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.247501 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerDied","Data":"b42de7c053d946fec107286630ff6cf5255040004852f11fd8d4558b15e24e1c"} Nov 03 17:00:05.247977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.247568 5946 scope.go:122] "RemoveContainer" containerID="b42e29b5d353925972c74cc3a69a1151c0406afe4a002e9a69d030722883a8ea" Nov 03 17:00:05.247977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.247755 5946 scope.go:122] "RemoveContainer" containerID="b42de7c053d946fec107286630ff6cf5255040004852f11fd8d4558b15e24e1c" Nov 03 17:00:05.248163 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.248077 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm_odh-ai-gateway-infra(19311f79-4585-4ddf-894c-a3ba0577d9bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" Nov 03 17:00:05.249371 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.249351 5946 generic.go:356] "Generic (PLEG): container finished" podID="6f797658-76a2-4569-b387-49b0680f7975" containerID="3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580" exitCode=0 Nov 03 17:00:05.249455 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.249409 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" Nov 03 17:00:05.249455 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.249435 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" event={"ID":"6f797658-76a2-4569-b387-49b0680f7975","Type":"ContainerDied","Data":"3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580"} Nov 03 17:00:05.249589 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.249493 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx" event={"ID":"6f797658-76a2-4569-b387-49b0680f7975","Type":"ContainerDied","Data":"67b710931d34e0b5849a5148b68be5600ab76987a2f948b5652414aa898dc8f8"} Nov 03 17:00:05.251510 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.251488 5946 scope.go:122] "RemoveContainer" containerID="f40e1064f3c84404f5f8c66fc76a95b416f8552c51a2409511cfae8bda4b56be" Nov 03 17:00:05.251758 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.251736 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb_odh-ai-gateway-infra(1dc28cee-8a22-4f6d-9f6a-48181d54e3ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" Nov 03 17:00:05.253033 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.253018 5946 scope.go:122] "RemoveContainer" containerID="9902906489fdf3c1e6d6aa61aba6a78f69bd71a99867c90ac82575117f169472" Nov 03 17:00:05.253254 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.253198 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895420-n9f6s_odh-ai-gateway-infra(32bdb396-45dd-42ab-be4b-fb8cc7f946d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" Nov 03 17:00:05.254660 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.254640 5946 generic.go:356] "Generic (PLEG): container finished" podID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerID="2bfe8eeed626d5a04fae0b400fad96f34871ceab238a7042d37dd559dfe92e72" exitCode=60 Nov 03 17:00:05.254761 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.254704 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerDied","Data":"2bfe8eeed626d5a04fae0b400fad96f34871ceab238a7042d37dd559dfe92e72"} Nov 03 17:00:05.254938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.254923 5946 scope.go:122] "RemoveContainer" containerID="2bfe8eeed626d5a04fae0b400fad96f34871ceab238a7042d37dd559dfe92e72" Nov 03 17:00:05.255175 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.255154 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn_odh-ai-gateway-infra(42492a4a-b6b9-4f1e-91d1-dd508fa57852)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" Nov 03 17:00:05.256467 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.256449 5946 generic.go:356] "Generic (PLEG): container finished" podID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerID="2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775" exitCode=60 Nov 03 17:00:05.256557 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.256548 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerDied","Data":"2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775"} Nov 03 17:00:05.256727 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.256712 5946 scope.go:122] "RemoveContainer" containerID="2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775" Nov 03 17:00:05.256917 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.256893 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln_odh-ai-gateway-infra(d396844f-cfd1-485c-a52f-33f6d944acd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" Nov 03 17:00:05.258517 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.258495 5946 generic.go:356] "Generic (PLEG): container finished" podID="7812793b-f556-46a7-8993-cfe7fa411179" containerID="b51acf8bb7989f028d6cb2222b1b3a62529767ad81034b2729335ba0e42a212f" exitCode=60 Nov 03 17:00:05.258593 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.258530 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerDied","Data":"b51acf8bb7989f028d6cb2222b1b3a62529767ad81034b2729335ba0e42a212f"} Nov 03 17:00:05.258735 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.258721 5946 scope.go:122] "RemoveContainer" containerID="b51acf8bb7989f028d6cb2222b1b3a62529767ad81034b2729335ba0e42a212f" Nov 03 17:00:05.258906 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.258890 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f_odh-ai-gateway-infra(7812793b-f556-46a7-8993-cfe7fa411179)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" podUID="7812793b-f556-46a7-8993-cfe7fa411179" Nov 03 17:00:05.264137 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.264119 5946 scope.go:122] "RemoveContainer" containerID="3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580" Nov 03 17:00:05.274105 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.274081 5946 scope.go:122] "RemoveContainer" containerID="fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0" Nov 03 17:00:05.283860 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.283829 5946 scope.go:122] "RemoveContainer" containerID="3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580" Nov 03 17:00:05.284242 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.284189 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580\": container with ID starting with 3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580 not found: ID does not exist" containerID="3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580" Nov 03 17:00:05.284354 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.284252 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580"} err="failed to get container status \"3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580\": rpc error: code = NotFound desc = could not find container \"3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580\": container with ID starting with 3ab270bafda5b4a625c60c6732e7c18885fdb058318e4c2ea6be2935839d7580 not found: ID does not exist" Nov 03 17:00:05.284354 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.284273 5946 scope.go:122] "RemoveContainer" containerID="fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0" Nov 03 17:00:05.284577 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:05.284557 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0\": container with ID starting with fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0 not found: ID does not exist" containerID="fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0" Nov 03 17:00:05.284648 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.284587 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0"} err="failed to get container status \"fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0\": rpc error: code = NotFound desc = could not find container \"fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0\": container with ID starting with fdfa06b4bc18d2c92d872d3a20aec4fb9052ebe4b533125b1a7bf82ffe4aacb0 not found: ID does not exist" Nov 03 17:00:05.284648 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.284604 5946 scope.go:122] "RemoveContainer" containerID="96476b5735249835facff6334da45afcd1dcc1da90cc0fd148b4876a2d5f47c2" Nov 03 17:00:05.298786 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.298764 5946 scope.go:122] "RemoveContainer" containerID="0d36c1308f697238c1f0fbda9259563c812b251d98569e5ffac1629ae5f4e990" Nov 03 17:00:05.312109 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.312089 5946 scope.go:122] "RemoveContainer" containerID="7e0258e039dde4bf9a71cd5ad21c7ddee2918f76ad1d5f93760bb449fb2f6e32" Nov 03 17:00:05.392312 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.392267 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:05.962722 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.962688 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx"] Nov 03 17:00:05.968280 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:05.968250 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7e55c3-openshift-default-6cc9f48987-5ndfx"] Nov 03 17:00:06.005952 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:06.005915 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f797658-76a2-4569-b387-49b0680f7975" path="/var/lib/kubelet/pods/6f797658-76a2-4569-b387-49b0680f7975/volumes" Nov 03 17:00:06.266870 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:06.266845 5946 scope.go:122] "RemoveContainer" containerID="2bfe8eeed626d5a04fae0b400fad96f34871ceab238a7042d37dd559dfe92e72" Nov 03 17:00:06.267365 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:06.267170 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn_odh-ai-gateway-infra(42492a4a-b6b9-4f1e-91d1-dd508fa57852)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" Nov 03 17:00:06.268654 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:06.268630 5946 scope.go:122] "RemoveContainer" containerID="2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775" Nov 03 17:00:06.268926 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:06.268906 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln_odh-ai-gateway-infra(d396844f-cfd1-485c-a52f-33f6d944acd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" Nov 03 17:00:06.270247 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:06.270195 5946 scope.go:122] "RemoveContainer" containerID="b51acf8bb7989f028d6cb2222b1b3a62529767ad81034b2729335ba0e42a212f" Nov 03 17:00:06.270538 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:06.270517 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f_odh-ai-gateway-infra(7812793b-f556-46a7-8993-cfe7fa411179)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" podUID="7812793b-f556-46a7-8993-cfe7fa411179" Nov 03 17:00:06.271782 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:06.271762 5946 scope.go:122] "RemoveContainer" containerID="b42de7c053d946fec107286630ff6cf5255040004852f11fd8d4558b15e24e1c" Nov 03 17:00:06.272000 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:06.271981 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm_odh-ai-gateway-infra(19311f79-4585-4ddf-894c-a3ba0577d9bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" Nov 03 17:00:06.854756 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:06.854719 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:07.299992 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.299947 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk"] Nov 03 17:00:07.301252 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.301226 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ce64b281-c110-441c-90e5-eff0960988b1" containerName="revoke-keys" Nov 03 17:00:07.301252 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.301252 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce64b281-c110-441c-90e5-eff0960988b1" containerName="revoke-keys" Nov 03 17:00:07.301439 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.301312 5946 cpu_manager.go:414] "containerMap: removing container" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" Nov 03 17:00:07.301439 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.301320 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" Nov 03 17:00:07.301658 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.301640 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" Nov 03 17:00:07.301715 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.301679 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" Nov 03 17:00:07.301715 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.301693 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce64b281-c110-441c-90e5-eff0960988b1" containerName="revoke-keys" Nov 03 17:00:07.307962 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.307939 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.362543 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.362509 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk"] Nov 03 17:00:07.372487 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.372458 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:00:07.410088 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.410044 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjvl\" (UniqueName: \"kubernetes.io/projected/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-kube-api-access-jjjvl\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.410290 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.410104 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.410290 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.410265 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.468462 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.468429 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:00:07.511721 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.511682 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjvl\" (UniqueName: \"kubernetes.io/projected/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-kube-api-access-jjjvl\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.511871 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.511729 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.511871 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.511766 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.512403 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.512377 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.514232 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.514190 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.576541 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.576465 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjvl\" (UniqueName: \"kubernetes.io/projected/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-kube-api-access-jjjvl\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:07.622604 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:07.622570 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:08.021994 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:08.021971 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk"] Nov 03 17:00:08.024522 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:00:08.024489 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db99d74_bbaa_4255_a50f_334c0d4ca0d7.slice/crio-6d96c64b16568986b2fc3fc0099bedf7cb17d2d6166ee5f600254d4c01fe8de2 WatchSource:0}: Error finding container 6d96c64b16568986b2fc3fc0099bedf7cb17d2d6166ee5f600254d4c01fe8de2: Status 404 returned error can't find the container with id 6d96c64b16568986b2fc3fc0099bedf7cb17d2d6166ee5f600254d4c01fe8de2 Nov 03 17:00:08.290282 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:08.290171 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" event={"ID":"7db99d74-bbaa-4255-a50f-334c0d4ca0d7","Type":"ContainerStarted","Data":"6d96c64b16568986b2fc3fc0099bedf7cb17d2d6166ee5f600254d4c01fe8de2"} Nov 03 17:00:09.299253 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:09.299200 5946 generic.go:356] "Generic (PLEG): container finished" podID="7db99d74-bbaa-4255-a50f-334c0d4ca0d7" containerID="d39b569404955135d858dbcebbb69025cff30b69bf242368d2796ce938a978fd" exitCode=0 Nov 03 17:00:09.299665 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:09.299297 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" event={"ID":"7db99d74-bbaa-4255-a50f-334c0d4ca0d7","Type":"ContainerDied","Data":"d39b569404955135d858dbcebbb69025cff30b69bf242368d2796ce938a978fd"} Nov 03 17:00:10.409358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.409312 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:10.458536 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.458511 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:10.545454 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.545416 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-xks-ca\") pod \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " Nov 03 17:00:10.545656 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.545589 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-openshift-ca\") pod \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " Nov 03 17:00:10.545656 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.545647 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-kube-api-access-jjjvl\" (UniqueName: \"kubernetes.io/projected/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-kube-api-access-jjjvl\") pod \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\" (UID: \"7db99d74-bbaa-4255-a50f-334c0d4ca0d7\") " Nov 03 17:00:10.545972 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.545940 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-openshift-ca" pod "7db99d74-bbaa-4255-a50f-334c0d4ca0d7" (UID: "7db99d74-bbaa-4255-a50f-334c0d4ca0d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:10.547904 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.547879 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-xks-ca" pod "7db99d74-bbaa-4255-a50f-334c0d4ca0d7" (UID: "7db99d74-bbaa-4255-a50f-334c0d4ca0d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:10.548019 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.547971 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-kube-api-access-jjjvl" pod "7db99d74-bbaa-4255-a50f-334c0d4ca0d7" (UID: "7db99d74-bbaa-4255-a50f-334c0d4ca0d7"). InnerVolumeSpecName "kube-api-access-jjjvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:10.647565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.647479 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:10.647565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.647512 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjvl\" (UniqueName: \"kubernetes.io/projected/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-kube-api-access-jjjvl\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:10.647565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:10.647524 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db99d74-bbaa-4255-a50f-334c0d4ca0d7-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:11.315776 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:11.315746 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" Nov 03 17:00:11.315962 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:11.315782 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk" event={"ID":"7db99d74-bbaa-4255-a50f-334c0d4ca0d7","Type":"ContainerDied","Data":"6d96c64b16568986b2fc3fc0099bedf7cb17d2d6166ee5f600254d4c01fe8de2"} Nov 03 17:00:11.315962 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:11.315838 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d96c64b16568986b2fc3fc0099bedf7cb17d2d6166ee5f600254d4c01fe8de2" Nov 03 17:00:11.845228 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:11.845182 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh"] Nov 03 17:00:11.945067 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:11.945028 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd"] Nov 03 17:00:12.586532 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:12.586499 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:00:12.603841 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:12.603811 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:13.997632 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:13.997595 5946 scope.go:122] "RemoveContainer" containerID="9902906489fdf3c1e6d6aa61aba6a78f69bd71a99867c90ac82575117f169472" Nov 03 17:00:14.336793 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:14.336767 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" event={"ID":"32bdb396-45dd-42ab-be4b-fb8cc7f946d4","Type":"ContainerStarted","Data":"77993ada19fdf4d729a98895f3c02da9a8b884f5244ede7efb4143d4cb69daef"} Nov 03 17:00:14.998086 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:14.998056 5946 scope.go:122] "RemoveContainer" containerID="2bfe8eeed626d5a04fae0b400fad96f34871ceab238a7042d37dd559dfe92e72" Nov 03 17:00:15.345503 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.345468 5946 generic.go:356] "Generic (PLEG): container finished" podID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerID="77993ada19fdf4d729a98895f3c02da9a8b884f5244ede7efb4143d4cb69daef" exitCode=60 Nov 03 17:00:15.345666 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.345540 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" event={"ID":"32bdb396-45dd-42ab-be4b-fb8cc7f946d4","Type":"ContainerDied","Data":"77993ada19fdf4d729a98895f3c02da9a8b884f5244ede7efb4143d4cb69daef"} Nov 03 17:00:15.345666 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.345592 5946 scope.go:122] "RemoveContainer" containerID="9902906489fdf3c1e6d6aa61aba6a78f69bd71a99867c90ac82575117f169472" Nov 03 17:00:15.345981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.345964 5946 scope.go:122] "RemoveContainer" containerID="77993ada19fdf4d729a98895f3c02da9a8b884f5244ede7efb4143d4cb69daef" Nov 03 17:00:15.346316 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:15.346297 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895420-n9f6s_odh-ai-gateway-infra(32bdb396-45dd-42ab-be4b-fb8cc7f946d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" Nov 03 17:00:15.347506 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.347480 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerStarted","Data":"0c1b7267c24671ddbcd64b9b06858c4cbde95aefc6e05a33a565d51b4734e5a0"} Nov 03 17:00:15.373580 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.373549 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:15.472144 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.472105 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s"] Nov 03 17:00:15.997195 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.997164 5946 scope.go:122] "RemoveContainer" containerID="f40e1064f3c84404f5f8c66fc76a95b416f8552c51a2409511cfae8bda4b56be" Nov 03 17:00:15.997377 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:15.997316 5946 scope.go:122] "RemoveContainer" containerID="b51acf8bb7989f028d6cb2222b1b3a62529767ad81034b2729335ba0e42a212f" Nov 03 17:00:16.359091 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.359062 5946 generic.go:356] "Generic (PLEG): container finished" podID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerID="0c1b7267c24671ddbcd64b9b06858c4cbde95aefc6e05a33a565d51b4734e5a0" exitCode=60 Nov 03 17:00:16.359504 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.359134 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerDied","Data":"0c1b7267c24671ddbcd64b9b06858c4cbde95aefc6e05a33a565d51b4734e5a0"} Nov 03 17:00:16.359504 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.359194 5946 scope.go:122] "RemoveContainer" containerID="2bfe8eeed626d5a04fae0b400fad96f34871ceab238a7042d37dd559dfe92e72" Nov 03 17:00:16.359723 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.359703 5946 scope.go:122] "RemoveContainer" containerID="0c1b7267c24671ddbcd64b9b06858c4cbde95aefc6e05a33a565d51b4734e5a0" Nov 03 17:00:16.360053 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:16.360015 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn_odh-ai-gateway-infra(42492a4a-b6b9-4f1e-91d1-dd508fa57852)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" Nov 03 17:00:16.361467 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.361447 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerStarted","Data":"2fb0f5b47e533768296f2c4bf6628653e932354682b8f82a9a5f82cdff3f0fd7"} Nov 03 17:00:16.505241 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.505200 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:16.617590 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.617552 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-kube-api-access-qh7x5\" (UniqueName: \"kubernetes.io/projected/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-kube-api-access-qh7x5\") pod \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " Nov 03 17:00:16.617788 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.617692 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-maas-api-service-ca\") pod \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\" (UID: \"32bdb396-45dd-42ab-be4b-fb8cc7f946d4\") " Nov 03 17:00:16.618029 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.618005 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-maas-api-service-ca" pod "32bdb396-45dd-42ab-be4b-fb8cc7f946d4" (UID: "32bdb396-45dd-42ab-be4b-fb8cc7f946d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:16.619763 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.619739 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-kube-api-access-qh7x5" pod "32bdb396-45dd-42ab-be4b-fb8cc7f946d4" (UID: "32bdb396-45dd-42ab-be4b-fb8cc7f946d4"). InnerVolumeSpecName "kube-api-access-qh7x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:16.644480 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.644445 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn"] Nov 03 17:00:16.718740 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.718706 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:16.718740 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.718734 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7x5\" (UniqueName: \"kubernetes.io/projected/32bdb396-45dd-42ab-be4b-fb8cc7f946d4-kube-api-access-qh7x5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:16.845906 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.845851 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" podUID="e201e468-b064-47e2-9e65-6b566718d93f" containerName="payload-processing" containerID="cri-o://f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be" gracePeriod=25 Nov 03 17:00:16.945931 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:16.945898 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" podUID="68da1c92-b2af-4617-86e2-292d5194cdba" containerName="payload-pre-processing" containerID="cri-o://abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3" gracePeriod=25 Nov 03 17:00:17.106375 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.106349 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 17:00:17.223998 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.223970 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e201e468-b064-47e2-9e65-6b566718d93f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e201e468-b064-47e2-9e65-6b566718d93f-plugins-config-volume\") pod \"e201e468-b064-47e2-9e65-6b566718d93f\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " Nov 03 17:00:17.224332 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.224097 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e201e468-b064-47e2-9e65-6b566718d93f-kube-api-access-csfp9\" (UniqueName: \"kubernetes.io/projected/e201e468-b064-47e2-9e65-6b566718d93f-kube-api-access-csfp9\") pod \"e201e468-b064-47e2-9e65-6b566718d93f\" (UID: \"e201e468-b064-47e2-9e65-6b566718d93f\") " Nov 03 17:00:17.224438 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.224398 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e201e468-b064-47e2-9e65-6b566718d93f-plugins-config-volume" pod "e201e468-b064-47e2-9e65-6b566718d93f" (UID: "e201e468-b064-47e2-9e65-6b566718d93f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:17.224654 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.224628 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e201e468-b064-47e2-9e65-6b566718d93f-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:17.226576 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.226552 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e201e468-b064-47e2-9e65-6b566718d93f-kube-api-access-csfp9" pod "e201e468-b064-47e2-9e65-6b566718d93f" (UID: "e201e468-b064-47e2-9e65-6b566718d93f"). InnerVolumeSpecName "kube-api-access-csfp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:17.325707 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.325667 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfp9\" (UniqueName: \"kubernetes.io/projected/e201e468-b064-47e2-9e65-6b566718d93f-kube-api-access-csfp9\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:17.370749 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.370712 5946 generic.go:356] "Generic (PLEG): container finished" podID="e201e468-b064-47e2-9e65-6b566718d93f" containerID="f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be" exitCode=0 Nov 03 17:00:17.371183 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.370774 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" event={"ID":"e201e468-b064-47e2-9e65-6b566718d93f","Type":"ContainerDied","Data":"f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be"} Nov 03 17:00:17.371183 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.370823 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" event={"ID":"e201e468-b064-47e2-9e65-6b566718d93f","Type":"ContainerDied","Data":"832b2f7d96f4f49753d9be50b37c4ec0ccaf3eae3eb759825a6624e6c327ec80"} Nov 03 17:00:17.371183 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.370835 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh" Nov 03 17:00:17.371183 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.370849 5946 scope.go:122] "RemoveContainer" containerID="f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be" Nov 03 17:00:17.373000 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.372975 5946 generic.go:356] "Generic (PLEG): container finished" podID="7812793b-f556-46a7-8993-cfe7fa411179" containerID="2fb0f5b47e533768296f2c4bf6628653e932354682b8f82a9a5f82cdff3f0fd7" exitCode=60 Nov 03 17:00:17.373133 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.373007 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerDied","Data":"2fb0f5b47e533768296f2c4bf6628653e932354682b8f82a9a5f82cdff3f0fd7"} Nov 03 17:00:17.373598 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.373578 5946 scope.go:122] "RemoveContainer" containerID="2fb0f5b47e533768296f2c4bf6628653e932354682b8f82a9a5f82cdff3f0fd7" Nov 03 17:00:17.373911 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:17.373886 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f_odh-ai-gateway-infra(7812793b-f556-46a7-8993-cfe7fa411179)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" podUID="7812793b-f556-46a7-8993-cfe7fa411179" Nov 03 17:00:17.376009 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.375985 5946 generic.go:356] "Generic (PLEG): container finished" podID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerID="8ca460b89dcaa6fb9e8d6eb37f883f914102eaa7ea970d1ff782bdd66e147f14" exitCode=60 Nov 03 17:00:17.376115 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.376054 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" event={"ID":"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba","Type":"ContainerDied","Data":"8ca460b89dcaa6fb9e8d6eb37f883f914102eaa7ea970d1ff782bdd66e147f14"} Nov 03 17:00:17.376500 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.376481 5946 scope.go:122] "RemoveContainer" containerID="8ca460b89dcaa6fb9e8d6eb37f883f914102eaa7ea970d1ff782bdd66e147f14" Nov 03 17:00:17.376856 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:17.376801 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb_odh-ai-gateway-infra(1dc28cee-8a22-4f6d-9f6a-48181d54e3ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" Nov 03 17:00:17.377821 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.377804 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" Nov 03 17:00:17.377915 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.377825 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s" event={"ID":"32bdb396-45dd-42ab-be4b-fb8cc7f946d4","Type":"ContainerDied","Data":"297b998edad50a576dfc4146cc747335f1be7c09812f4a5775ba56c2cad70b70"} Nov 03 17:00:17.382758 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.382739 5946 scope.go:122] "RemoveContainer" containerID="f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be" Nov 03 17:00:17.383066 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:17.383049 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be\": container with ID starting with f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be not found: ID does not exist" containerID="f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be" Nov 03 17:00:17.383152 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.383072 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be"} err="failed to get container status \"f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be\": rpc error: code = NotFound desc = could not find container \"f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be\": container with ID starting with f0f528c0e4dcd77effacd6e05ced3de5fa5d57077165f7557e90ee81524717be not found: ID does not exist" Nov 03 17:00:17.383152 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.383086 5946 scope.go:122] "RemoveContainer" containerID="b51acf8bb7989f028d6cb2222b1b3a62529767ad81034b2729335ba0e42a212f" Nov 03 17:00:17.397761 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.397736 5946 scope.go:122] "RemoveContainer" containerID="f40e1064f3c84404f5f8c66fc76a95b416f8552c51a2409511cfae8bda4b56be" Nov 03 17:00:17.413319 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.413290 5946 scope.go:122] "RemoveContainer" containerID="77993ada19fdf4d729a98895f3c02da9a8b884f5244ede7efb4143d4cb69daef" Nov 03 17:00:17.531453 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.531428 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:17.619606 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.619523 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f"] Nov 03 17:00:17.628328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.628288 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42492a4a-b6b9-4f1e-91d1-dd508fa57852-kube-api-access-p9kh5\" (UniqueName: \"kubernetes.io/projected/42492a4a-b6b9-4f1e-91d1-dd508fa57852-kube-api-access-p9kh5\") pod \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " Nov 03 17:00:17.628489 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.628347 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42492a4a-b6b9-4f1e-91d1-dd508fa57852-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42492a4a-b6b9-4f1e-91d1-dd508fa57852-maas-api-service-ca\") pod \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\" (UID: \"42492a4a-b6b9-4f1e-91d1-dd508fa57852\") " Nov 03 17:00:17.628769 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.628739 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42492a4a-b6b9-4f1e-91d1-dd508fa57852-maas-api-service-ca" pod "42492a4a-b6b9-4f1e-91d1-dd508fa57852" (UID: "42492a4a-b6b9-4f1e-91d1-dd508fa57852"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:17.630668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.630648 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42492a4a-b6b9-4f1e-91d1-dd508fa57852-kube-api-access-p9kh5" pod "42492a4a-b6b9-4f1e-91d1-dd508fa57852" (UID: "42492a4a-b6b9-4f1e-91d1-dd508fa57852"). InnerVolumeSpecName "kube-api-access-p9kh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:17.729835 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.729802 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9kh5\" (UniqueName: \"kubernetes.io/projected/42492a4a-b6b9-4f1e-91d1-dd508fa57852-kube-api-access-p9kh5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:17.729835 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.729835 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42492a4a-b6b9-4f1e-91d1-dd508fa57852-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:17.730289 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.730235 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s"] Nov 03 17:00:17.741192 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.741163 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-n9f6s"] Nov 03 17:00:17.977223 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.977097 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh"] Nov 03 17:00:17.984597 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:17.984556 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61be78-74668f598f-9bqnh"] Nov 03 17:00:18.003485 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.003448 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" path="/var/lib/kubelet/pods/32bdb396-45dd-42ab-be4b-fb8cc7f946d4/volumes" Nov 03 17:00:18.003871 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.003858 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e201e468-b064-47e2-9e65-6b566718d93f" path="/var/lib/kubelet/pods/e201e468-b064-47e2-9e65-6b566718d93f/volumes" Nov 03 17:00:18.390703 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.390673 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" event={"ID":"42492a4a-b6b9-4f1e-91d1-dd508fa57852","Type":"ContainerDied","Data":"c3dbf3a780f5c8ebec4307ab07b5597b640b94bfe9d796ab77160944ac00fa4f"} Nov 03 17:00:18.390703 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.390695 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn" Nov 03 17:00:18.391172 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.390739 5946 scope.go:122] "RemoveContainer" containerID="0c1b7267c24671ddbcd64b9b06858c4cbde95aefc6e05a33a565d51b4734e5a0" Nov 03 17:00:18.532441 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.532417 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:18.624868 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.624830 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb"] Nov 03 17:00:18.627981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.627952 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn"] Nov 03 17:00:18.630778 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.630757 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c88d54-29895420-fz8pn"] Nov 03 17:00:18.642007 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.641943 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7812793b-f556-46a7-8993-cfe7fa411179-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812793b-f556-46a7-8993-cfe7fa411179-maas-api-service-ca\") pod \"7812793b-f556-46a7-8993-cfe7fa411179\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " Nov 03 17:00:18.642117 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.642064 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7812793b-f556-46a7-8993-cfe7fa411179-kube-api-access-jx68p\" (UniqueName: \"kubernetes.io/projected/7812793b-f556-46a7-8993-cfe7fa411179-kube-api-access-jx68p\") pod \"7812793b-f556-46a7-8993-cfe7fa411179\" (UID: \"7812793b-f556-46a7-8993-cfe7fa411179\") " Nov 03 17:00:18.642383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.642360 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7812793b-f556-46a7-8993-cfe7fa411179-maas-api-service-ca" pod "7812793b-f556-46a7-8993-cfe7fa411179" (UID: "7812793b-f556-46a7-8993-cfe7fa411179"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:18.642491 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.642478 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812793b-f556-46a7-8993-cfe7fa411179-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:18.644391 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.644365 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7812793b-f556-46a7-8993-cfe7fa411179-kube-api-access-jx68p" pod "7812793b-f556-46a7-8993-cfe7fa411179" (UID: "7812793b-f556-46a7-8993-cfe7fa411179"). InnerVolumeSpecName "kube-api-access-jx68p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.736731 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.736699 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z"] Nov 03 17:00:18.743270 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.743242 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx68p\" (UniqueName: \"kubernetes.io/projected/7812793b-f556-46a7-8993-cfe7fa411179-kube-api-access-jx68p\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:18.768438 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.768414 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:18.838626 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.838592 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw"] Nov 03 17:00:18.945230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.945128 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-kube-api-access-6m55l\" (UniqueName: \"kubernetes.io/projected/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-kube-api-access-6m55l\") pod \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " Nov 03 17:00:18.945417 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.945283 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-maas-api-service-ca\") pod \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\" (UID: \"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba\") " Nov 03 17:00:18.945636 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.945614 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-maas-api-service-ca" pod "1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" (UID: "1dc28cee-8a22-4f6d-9f6a-48181d54e3ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:18.945731 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.945701 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:18.947442 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.947422 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-kube-api-access-6m55l" pod "1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" (UID: "1dc28cee-8a22-4f6d-9f6a-48181d54e3ba"). InnerVolumeSpecName "kube-api-access-6m55l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.997499 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.997464 5946 scope.go:122] "RemoveContainer" containerID="b42de7c053d946fec107286630ff6cf5255040004852f11fd8d4558b15e24e1c" Nov 03 17:00:18.997668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:18.997575 5946 scope.go:122] "RemoveContainer" containerID="2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775" Nov 03 17:00:19.047130 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.047099 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m55l\" (UniqueName: \"kubernetes.io/projected/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba-kube-api-access-6m55l\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:19.139896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.139844 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:19.336645 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.336620 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:19.402370 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.402339 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerStarted","Data":"fb7d8002eb7c2bbb0921cbe7c465c8d558a49a89220fc20ef37fc5500e03b3b2"} Nov 03 17:00:19.403696 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.403667 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" Nov 03 17:00:19.403696 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.403678 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f" event={"ID":"7812793b-f556-46a7-8993-cfe7fa411179","Type":"ContainerDied","Data":"0381854cdce5a32099644bf3d49bfc9f8e0fe99bfd958d344ae9fed5c5573840"} Nov 03 17:00:19.403892 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.403736 5946 scope.go:122] "RemoveContainer" containerID="2fb0f5b47e533768296f2c4bf6628653e932354682b8f82a9a5f82cdff3f0fd7" Nov 03 17:00:19.405658 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.405637 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerStarted","Data":"0f2303cd06cea1ce16efa1352ce5b5b6d60b34530041c0f981d7b4964c4e6429"} Nov 03 17:00:19.406939 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.406916 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" Nov 03 17:00:19.406939 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.406927 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb" event={"ID":"1dc28cee-8a22-4f6d-9f6a-48181d54e3ba","Type":"ContainerDied","Data":"ec061283f7b8f9e9ff1f429b39bba50f541620adbf296530972585804b0eb84a"} Nov 03 17:00:19.414949 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.414928 5946 scope.go:122] "RemoveContainer" containerID="8ca460b89dcaa6fb9e8d6eb37f883f914102eaa7ea970d1ff782bdd66e147f14" Nov 03 17:00:19.641369 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.641331 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f"] Nov 03 17:00:19.649351 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.649324 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bcd756-29895420-2n67f"] Nov 03 17:00:19.763918 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.763884 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb"] Nov 03 17:00:19.770450 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:19.770417 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9f2868-29895420-bp9vb"] Nov 03 17:00:20.004693 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.004661 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" path="/var/lib/kubelet/pods/1dc28cee-8a22-4f6d-9f6a-48181d54e3ba/volumes" Nov 03 17:00:20.005190 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.005172 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" path="/var/lib/kubelet/pods/42492a4a-b6b9-4f1e-91d1-dd508fa57852/volumes" Nov 03 17:00:20.005650 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.005636 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7812793b-f556-46a7-8993-cfe7fa411179" path="/var/lib/kubelet/pods/7812793b-f556-46a7-8993-cfe7fa411179/volumes" Nov 03 17:00:20.256379 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.256343 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:20.416417 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.416378 5946 generic.go:356] "Generic (PLEG): container finished" podID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerID="0f2303cd06cea1ce16efa1352ce5b5b6d60b34530041c0f981d7b4964c4e6429" exitCode=60 Nov 03 17:00:20.416886 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.416447 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerDied","Data":"0f2303cd06cea1ce16efa1352ce5b5b6d60b34530041c0f981d7b4964c4e6429"} Nov 03 17:00:20.416886 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.416514 5946 scope.go:122] "RemoveContainer" containerID="b42de7c053d946fec107286630ff6cf5255040004852f11fd8d4558b15e24e1c" Nov 03 17:00:20.417023 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.416916 5946 scope.go:122] "RemoveContainer" containerID="0f2303cd06cea1ce16efa1352ce5b5b6d60b34530041c0f981d7b4964c4e6429" Nov 03 17:00:20.417346 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:20.417318 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm_odh-ai-gateway-infra(19311f79-4585-4ddf-894c-a3ba0577d9bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" Nov 03 17:00:20.426751 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.426688 5946 generic.go:356] "Generic (PLEG): container finished" podID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerID="fb7d8002eb7c2bbb0921cbe7c465c8d558a49a89220fc20ef37fc5500e03b3b2" exitCode=60 Nov 03 17:00:20.426751 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.426748 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerDied","Data":"fb7d8002eb7c2bbb0921cbe7c465c8d558a49a89220fc20ef37fc5500e03b3b2"} Nov 03 17:00:20.427680 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.427199 5946 scope.go:122] "RemoveContainer" containerID="fb7d8002eb7c2bbb0921cbe7c465c8d558a49a89220fc20ef37fc5500e03b3b2" Nov 03 17:00:20.427680 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:20.427546 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln_odh-ai-gateway-infra(d396844f-cfd1-485c-a52f-33f6d944acd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" Nov 03 17:00:20.442413 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.442384 5946 scope.go:122] "RemoveContainer" containerID="2e690b1673685fb830673cedd920f784506f78804619fbadc4153e263c569775" Nov 03 17:00:20.535857 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.535758 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln"] Nov 03 17:00:20.904692 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:20.904601 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm"] Nov 03 17:00:21.618305 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.618282 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:21.620930 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.620912 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:21.781668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.781637 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d396844f-cfd1-485c-a52f-33f6d944acd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d396844f-cfd1-485c-a52f-33f6d944acd1-maas-api-service-ca\") pod \"d396844f-cfd1-485c-a52f-33f6d944acd1\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " Nov 03 17:00:21.781837 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.781715 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19311f79-4585-4ddf-894c-a3ba0577d9bf-kube-api-access-x72tk\" (UniqueName: \"kubernetes.io/projected/19311f79-4585-4ddf-894c-a3ba0577d9bf-kube-api-access-x72tk\") pod \"19311f79-4585-4ddf-894c-a3ba0577d9bf\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " Nov 03 17:00:21.781837 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.781735 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d396844f-cfd1-485c-a52f-33f6d944acd1-kube-api-access-cpcgr\" (UniqueName: \"kubernetes.io/projected/d396844f-cfd1-485c-a52f-33f6d944acd1-kube-api-access-cpcgr\") pod \"d396844f-cfd1-485c-a52f-33f6d944acd1\" (UID: \"d396844f-cfd1-485c-a52f-33f6d944acd1\") " Nov 03 17:00:21.781963 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.781844 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19311f79-4585-4ddf-894c-a3ba0577d9bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19311f79-4585-4ddf-894c-a3ba0577d9bf-maas-api-service-ca\") pod \"19311f79-4585-4ddf-894c-a3ba0577d9bf\" (UID: \"19311f79-4585-4ddf-894c-a3ba0577d9bf\") " Nov 03 17:00:21.782060 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.782033 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d396844f-cfd1-485c-a52f-33f6d944acd1-maas-api-service-ca" pod "d396844f-cfd1-485c-a52f-33f6d944acd1" (UID: "d396844f-cfd1-485c-a52f-33f6d944acd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:21.782260 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.782239 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d396844f-cfd1-485c-a52f-33f6d944acd1-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:21.782356 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.782287 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19311f79-4585-4ddf-894c-a3ba0577d9bf-maas-api-service-ca" pod "19311f79-4585-4ddf-894c-a3ba0577d9bf" (UID: "19311f79-4585-4ddf-894c-a3ba0577d9bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:21.784005 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.783985 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d396844f-cfd1-485c-a52f-33f6d944acd1-kube-api-access-cpcgr" pod "d396844f-cfd1-485c-a52f-33f6d944acd1" (UID: "d396844f-cfd1-485c-a52f-33f6d944acd1"). InnerVolumeSpecName "kube-api-access-cpcgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:21.784089 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.784069 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19311f79-4585-4ddf-894c-a3ba0577d9bf-kube-api-access-x72tk" pod "19311f79-4585-4ddf-894c-a3ba0577d9bf" (UID: "19311f79-4585-4ddf-894c-a3ba0577d9bf"). InnerVolumeSpecName "kube-api-access-x72tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:21.882838 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.882809 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72tk\" (UniqueName: \"kubernetes.io/projected/19311f79-4585-4ddf-894c-a3ba0577d9bf-kube-api-access-x72tk\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:21.882838 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.882835 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpcgr\" (UniqueName: \"kubernetes.io/projected/d396844f-cfd1-485c-a52f-33f6d944acd1-kube-api-access-cpcgr\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:21.883044 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:21.882845 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19311f79-4585-4ddf-894c-a3ba0577d9bf-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:22.445371 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.445342 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" Nov 03 17:00:22.445572 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.445375 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln" event={"ID":"d396844f-cfd1-485c-a52f-33f6d944acd1","Type":"ContainerDied","Data":"21114e42ac17f59dbbfb3556be42ebbfafad5e6819ac7a143270d8d02a56cb14"} Nov 03 17:00:22.445572 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.445443 5946 scope.go:122] "RemoveContainer" containerID="fb7d8002eb7c2bbb0921cbe7c465c8d558a49a89220fc20ef37fc5500e03b3b2" Nov 03 17:00:22.446996 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.446960 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" event={"ID":"19311f79-4585-4ddf-894c-a3ba0577d9bf","Type":"ContainerDied","Data":"efac5f7e34b8ceab11b4664f2d4323a3d1bcf97db4bf5d4cbd352c68d8094389"} Nov 03 17:00:22.446996 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.446987 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm" Nov 03 17:00:22.455432 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.455412 5946 scope.go:122] "RemoveContainer" containerID="0f2303cd06cea1ce16efa1352ce5b5b6d60b34530041c0f981d7b4964c4e6429" Nov 03 17:00:22.619137 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.619096 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm"] Nov 03 17:00:22.670323 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.670288 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895420-pgdlm"] Nov 03 17:00:22.741691 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.741653 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln"] Nov 03 17:00:22.744618 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:22.744595 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-195095-29895420-2z4ln"] Nov 03 17:00:23.737700 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:23.737662 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" podUID="b36bb1be-b86c-4269-a34c-5d6c2534c102" containerName="payload-processing" containerID="cri-o://fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8" gracePeriod=25 Nov 03 17:00:23.839721 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:23.839673 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" podUID="e450076f-d096-468a-8607-cc46d61d1c7a" containerName="payload-pre-processing" containerID="cri-o://8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456" gracePeriod=25 Nov 03 17:00:24.001050 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.000982 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 17:00:24.004217 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.004176 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" path="/var/lib/kubelet/pods/19311f79-4585-4ddf-894c-a3ba0577d9bf/volumes" Nov 03 17:00:24.004636 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.004620 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" path="/var/lib/kubelet/pods/d396844f-cfd1-485c-a52f-33f6d944acd1/volumes" Nov 03 17:00:24.031895 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.031862 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" podUID="68da1c92-b2af-4617-86e2-292d5194cdba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:00:24.102617 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.102582 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36bb1be-b86c-4269-a34c-5d6c2534c102-kube-api-access-hkssh\" (UniqueName: \"kubernetes.io/projected/b36bb1be-b86c-4269-a34c-5d6c2534c102-kube-api-access-hkssh\") pod \"b36bb1be-b86c-4269-a34c-5d6c2534c102\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " Nov 03 17:00:24.102790 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.102692 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36bb1be-b86c-4269-a34c-5d6c2534c102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36bb1be-b86c-4269-a34c-5d6c2534c102-plugins-config-volume\") pod \"b36bb1be-b86c-4269-a34c-5d6c2534c102\" (UID: \"b36bb1be-b86c-4269-a34c-5d6c2534c102\") " Nov 03 17:00:24.103079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.103055 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36bb1be-b86c-4269-a34c-5d6c2534c102-plugins-config-volume" pod "b36bb1be-b86c-4269-a34c-5d6c2534c102" (UID: "b36bb1be-b86c-4269-a34c-5d6c2534c102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:24.104882 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.104861 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36bb1be-b86c-4269-a34c-5d6c2534c102-kube-api-access-hkssh" pod "b36bb1be-b86c-4269-a34c-5d6c2534c102" (UID: "b36bb1be-b86c-4269-a34c-5d6c2534c102"). InnerVolumeSpecName "kube-api-access-hkssh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:24.203626 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.203587 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkssh\" (UniqueName: \"kubernetes.io/projected/b36bb1be-b86c-4269-a34c-5d6c2534c102-kube-api-access-hkssh\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:24.203626 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.203617 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36bb1be-b86c-4269-a34c-5d6c2534c102-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:24.463316 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.463193 5946 generic.go:356] "Generic (PLEG): container finished" podID="b36bb1be-b86c-4269-a34c-5d6c2534c102" containerID="fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8" exitCode=0 Nov 03 17:00:24.463316 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.463257 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" event={"ID":"b36bb1be-b86c-4269-a34c-5d6c2534c102","Type":"ContainerDied","Data":"fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8"} Nov 03 17:00:24.463316 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.463276 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" Nov 03 17:00:24.463578 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.463322 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z" event={"ID":"b36bb1be-b86c-4269-a34c-5d6c2534c102","Type":"ContainerDied","Data":"8ea16022ee517c14a4fe42b60eb75945ad18e409493e52c0d2b84506d214d155"} Nov 03 17:00:24.463578 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.463355 5946 scope.go:122] "RemoveContainer" containerID="fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8" Nov 03 17:00:24.474178 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.474158 5946 scope.go:122] "RemoveContainer" containerID="fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8" Nov 03 17:00:24.474556 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:24.474533 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8\": container with ID starting with fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8 not found: ID does not exist" containerID="fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8" Nov 03 17:00:24.474636 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.474563 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8"} err="failed to get container status \"fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8\": rpc error: code = NotFound desc = could not find container \"fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8\": container with ID starting with fdf1ac51fa9f11612497e8cbf3d1b0a36369b36158424eccef3e5a0cbe9774c8 not found: ID does not exist" Nov 03 17:00:24.635291 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.635252 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z"] Nov 03 17:00:24.686436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:24.686402 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf492-7679986d75-c4f8z"] Nov 03 17:00:25.063535 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:25.063497 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:26.003741 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:26.003701 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36bb1be-b86c-4269-a34c-5d6c2534c102" path="/var/lib/kubelet/pods/b36bb1be-b86c-4269-a34c-5d6c2534c102/volumes" Nov 03 17:00:26.737450 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:26.737418 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:26.929015 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:26.928969 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:29.766339 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:29.766295 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:31.842154 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:31.842112 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" podUID="e450076f-d096-468a-8607-cc46d61d1c7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:00:34.032304 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:34.032265 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" podUID="68da1c92-b2af-4617-86e2-292d5194cdba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:00:34.461807 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:34.461701 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:36.922458 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:36.922421 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:00:37.198401 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.198374 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 17:00:37.244918 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.244887 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68da1c92-b2af-4617-86e2-292d5194cdba-kube-api-access-54j4b\" (UniqueName: \"kubernetes.io/projected/68da1c92-b2af-4617-86e2-292d5194cdba-kube-api-access-54j4b\") pod \"68da1c92-b2af-4617-86e2-292d5194cdba\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " Nov 03 17:00:37.245079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.245011 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68da1c92-b2af-4617-86e2-292d5194cdba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68da1c92-b2af-4617-86e2-292d5194cdba-plugins-config-volume\") pod \"68da1c92-b2af-4617-86e2-292d5194cdba\" (UID: \"68da1c92-b2af-4617-86e2-292d5194cdba\") " Nov 03 17:00:37.245409 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.245386 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68da1c92-b2af-4617-86e2-292d5194cdba-plugins-config-volume" pod "68da1c92-b2af-4617-86e2-292d5194cdba" (UID: "68da1c92-b2af-4617-86e2-292d5194cdba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:37.247094 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.247071 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68da1c92-b2af-4617-86e2-292d5194cdba-kube-api-access-54j4b" pod "68da1c92-b2af-4617-86e2-292d5194cdba" (UID: "68da1c92-b2af-4617-86e2-292d5194cdba"). InnerVolumeSpecName "kube-api-access-54j4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:37.346265 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.346231 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54j4b\" (UniqueName: \"kubernetes.io/projected/68da1c92-b2af-4617-86e2-292d5194cdba-kube-api-access-54j4b\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:37.346265 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.346268 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68da1c92-b2af-4617-86e2-292d5194cdba-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:37.559970 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.559927 5946 generic.go:356] "Generic (PLEG): container finished" podID="68da1c92-b2af-4617-86e2-292d5194cdba" containerID="abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3" exitCode=0 Nov 03 17:00:37.560188 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.560089 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" Nov 03 17:00:37.560188 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.560098 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" event={"ID":"68da1c92-b2af-4617-86e2-292d5194cdba","Type":"ContainerDied","Data":"abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3"} Nov 03 17:00:37.560188 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.560170 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd" event={"ID":"68da1c92-b2af-4617-86e2-292d5194cdba","Type":"ContainerDied","Data":"c01ad1270dbdbc3b11e9aa8ea879d92a57ce47820a4e55153ed9a5507a5878d9"} Nov 03 17:00:37.560396 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.560331 5946 scope.go:122] "RemoveContainer" containerID="abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3" Nov 03 17:00:37.571262 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.571243 5946 scope.go:122] "RemoveContainer" containerID="abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3" Nov 03 17:00:37.571548 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:37.571527 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3\": container with ID starting with abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3 not found: ID does not exist" containerID="abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3" Nov 03 17:00:37.571624 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.571559 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3"} err="failed to get container status \"abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3\": rpc error: code = NotFound desc = could not find container \"abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3\": container with ID starting with abd74fdc12249005585e5eba6dd257aaf20fee39f7736beca49632083dd01da3 not found: ID does not exist" Nov 03 17:00:37.734069 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.734026 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd"] Nov 03 17:00:37.785528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:37.785485 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61be78-6d6556dcc5-fjvgd"] Nov 03 17:00:38.003833 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:38.003799 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68da1c92-b2af-4617-86e2-292d5194cdba" path="/var/lib/kubelet/pods/68da1c92-b2af-4617-86e2-292d5194cdba/volumes" Nov 03 17:00:39.261150 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:39.261107 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:40.887556 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:40.887505 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:41.842508 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:41.842473 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" podUID="e450076f-d096-468a-8607-cc46d61d1c7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:00:42.586361 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:42.586326 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:00:43.382994 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:43.382962 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:44.094108 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.094083 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 17:00:44.221322 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.221285 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e450076f-d096-468a-8607-cc46d61d1c7a-kube-api-access-t9spb\" (UniqueName: \"kubernetes.io/projected/e450076f-d096-468a-8607-cc46d61d1c7a-kube-api-access-t9spb\") pod \"e450076f-d096-468a-8607-cc46d61d1c7a\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " Nov 03 17:00:44.221518 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.221346 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e450076f-d096-468a-8607-cc46d61d1c7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e450076f-d096-468a-8607-cc46d61d1c7a-plugins-config-volume\") pod \"e450076f-d096-468a-8607-cc46d61d1c7a\" (UID: \"e450076f-d096-468a-8607-cc46d61d1c7a\") " Nov 03 17:00:44.221716 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.221692 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e450076f-d096-468a-8607-cc46d61d1c7a-plugins-config-volume" pod "e450076f-d096-468a-8607-cc46d61d1c7a" (UID: "e450076f-d096-468a-8607-cc46d61d1c7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:44.221800 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.221787 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e450076f-d096-468a-8607-cc46d61d1c7a-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:44.223929 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.223896 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e450076f-d096-468a-8607-cc46d61d1c7a-kube-api-access-t9spb" pod "e450076f-d096-468a-8607-cc46d61d1c7a" (UID: "e450076f-d096-468a-8607-cc46d61d1c7a"). InnerVolumeSpecName "kube-api-access-t9spb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:44.322548 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.322468 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9spb\" (UniqueName: \"kubernetes.io/projected/e450076f-d096-468a-8607-cc46d61d1c7a-kube-api-access-t9spb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:44.503141 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.503101 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:44.615609 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.615357 5946 generic.go:356] "Generic (PLEG): container finished" podID="e450076f-d096-468a-8607-cc46d61d1c7a" containerID="8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456" exitCode=0 Nov 03 17:00:44.615609 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.615423 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" Nov 03 17:00:44.615609 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.615449 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" event={"ID":"e450076f-d096-468a-8607-cc46d61d1c7a","Type":"ContainerDied","Data":"8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456"} Nov 03 17:00:44.615609 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.615514 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw" event={"ID":"e450076f-d096-468a-8607-cc46d61d1c7a","Type":"ContainerDied","Data":"f2a7d52d57a603a84f6a1a39010363c9cc8b9f59f630d23bc511f2373b722aca"} Nov 03 17:00:44.615609 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.615546 5946 scope.go:122] "RemoveContainer" containerID="8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456" Nov 03 17:00:44.626764 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.626742 5946 scope.go:122] "RemoveContainer" containerID="8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456" Nov 03 17:00:44.627052 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:00:44.627031 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456\": container with ID starting with 8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456 not found: ID does not exist" containerID="8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456" Nov 03 17:00:44.627137 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.627060 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456"} err="failed to get container status \"8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456\": rpc error: code = NotFound desc = could not find container \"8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456\": container with ID starting with 8e6531dc2eb529b97084d442eabcf3cc0aead4988296c5d0be087bb4e2aaf456 not found: ID does not exist" Nov 03 17:00:44.803164 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.803126 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw"] Nov 03 17:00:44.849051 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.849006 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:44.852086 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:44.852058 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf492-7688654fc9-cmjgw"] Nov 03 17:00:45.121515 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:45.121472 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:45.349091 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:45.349055 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2"] Nov 03 17:00:45.352135 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:45.352092 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5937b-82pf2"] Nov 03 17:00:45.649298 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:45.649245 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:45.850383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:45.850347 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:46.004762 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:46.004725 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e450076f-d096-468a-8607-cc46d61d1c7a" path="/var/lib/kubelet/pods/e450076f-d096-468a-8607-cc46d61d1c7a/volumes" Nov 03 17:00:46.005151 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:46.005137 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4" path="/var/lib/kubelet/pods/e92cb43e-64e5-4b3d-b5c8-d16b7df2a7b4/volumes" Nov 03 17:00:46.054394 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:46.054360 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:46.355733 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:46.355635 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:46.568080 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:46.568042 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk"] Nov 03 17:00:46.618716 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:46.618626 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf492-l5dxk"] Nov 03 17:00:47.309761 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:47.309721 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:47.549254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:47.549221 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:47.742331 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:47.742296 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:47.814445 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:47.814407 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln"] Nov 03 17:00:47.865233 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:47.865185 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61be78-q4fln"] Nov 03 17:00:48.003806 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:48.003725 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db99d74-bbaa-4255-a50f-334c0d4ca0d7" path="/var/lib/kubelet/pods/7db99d74-bbaa-4255-a50f-334c0d4ca0d7/volumes" Nov 03 17:00:48.004250 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:48.004235 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce64b281-c110-441c-90e5-eff0960988b1" path="/var/lib/kubelet/pods/ce64b281-c110-441c-90e5-eff0960988b1/volumes" Nov 03 17:00:48.308652 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:48.308558 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:52.610476 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:52.610433 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:52.835904 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:52.835875 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:53.050950 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:53.050917 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:53.248807 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:53.248771 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:54.101119 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101082 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm"] Nov 03 17:00:54.101582 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101567 5946 cpu_manager.go:414] "containerMap: removing container" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101583 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101593 5946 cpu_manager.go:414] "containerMap: removing container" podUID="68da1c92-b2af-4617-86e2-292d5194cdba" containerName="payload-pre-processing" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101599 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68da1c92-b2af-4617-86e2-292d5194cdba" containerName="payload-pre-processing" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101607 5946 cpu_manager.go:414] "containerMap: removing container" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101611 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101618 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101622 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101634 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:54.101635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101639 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101648 5946 cpu_manager.go:414] "containerMap: removing container" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101652 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101661 5946 cpu_manager.go:414] "containerMap: removing container" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101665 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101671 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101675 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101682 5946 cpu_manager.go:414] "containerMap: removing container" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101686 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101698 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101703 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101710 5946 cpu_manager.go:414] "containerMap: removing container" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101714 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f797658-76a2-4569-b387-49b0680f7975" containerName="istio-proxy" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101721 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b36bb1be-b86c-4269-a34c-5d6c2534c102" containerName="payload-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101725 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36bb1be-b86c-4269-a34c-5d6c2534c102" containerName="payload-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101731 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e201e468-b064-47e2-9e65-6b566718d93f" containerName="payload-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101735 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e201e468-b064-47e2-9e65-6b566718d93f" containerName="payload-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101741 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7db99d74-bbaa-4255-a50f-334c0d4ca0d7" containerName="revoke-keys" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101745 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db99d74-bbaa-4255-a50f-334c0d4ca0d7" containerName="revoke-keys" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101754 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101758 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101765 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101769 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101778 5946 cpu_manager.go:414] "containerMap: removing container" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101783 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101796 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101800 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101806 5946 cpu_manager.go:414] "containerMap: removing container" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101812 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101823 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e450076f-d096-468a-8607-cc46d61d1c7a" containerName="payload-pre-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101828 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e450076f-d096-468a-8607-cc46d61d1c7a" containerName="payload-pre-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101891 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101899 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101905 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e450076f-d096-468a-8607-cc46d61d1c7a" containerName="payload-pre-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101914 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101919 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="68da1c92-b2af-4617-86e2-292d5194cdba" containerName="payload-pre-processing" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101926 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101930 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101936 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101941 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:54.101926 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101946 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101952 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101957 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101962 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db99d74-bbaa-4255-a50f-334c0d4ca0d7" containerName="revoke-keys" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101966 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36bb1be-b86c-4269-a34c-5d6c2534c102" containerName="payload-processing" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101971 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e201e468-b064-47e2-9e65-6b566718d93f" containerName="payload-processing" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101977 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101981 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101986 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101992 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.101999 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7812793b-f556-46a7-8993-cfe7fa411179" containerName="cleanup" Nov 03 17:00:54.103159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.102003 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:54.106696 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.106679 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.163370 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.163340 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:00:54.166131 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.166100 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm"] Nov 03 17:00:54.166689 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.166662 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:00:54.214056 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.214022 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:00:54.214409 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.214391 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:00:54.223606 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.223580 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47230282-26de-49a2-83b0-29a86b164f20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.223728 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.223616 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47230282-26de-49a2-83b0-29a86b164f20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.223797 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.223783 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbqv\" (UniqueName: \"kubernetes.io/projected/47230282-26de-49a2-83b0-29a86b164f20-kube-api-access-4sbqv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.324742 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.324705 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47230282-26de-49a2-83b0-29a86b164f20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.324950 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.324866 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbqv\" (UniqueName: \"kubernetes.io/projected/47230282-26de-49a2-83b0-29a86b164f20-kube-api-access-4sbqv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.324950 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.324941 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47230282-26de-49a2-83b0-29a86b164f20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.325603 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.325584 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47230282-26de-49a2-83b0-29a86b164f20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.327361 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.327343 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47230282-26de-49a2-83b0-29a86b164f20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.386549 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.386470 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbqv\" (UniqueName: \"kubernetes.io/projected/47230282-26de-49a2-83b0-29a86b164f20-kube-api-access-4sbqv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.420365 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.420332 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:54.814645 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:54.811088 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm"] Nov 03 17:00:55.710242 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:55.710195 5946 generic.go:356] "Generic (PLEG): container finished" podID="47230282-26de-49a2-83b0-29a86b164f20" containerID="5a9a1cbd38efa5efdf9300246f1ffe4152741ba55922fc7adbc2fe82ad0dde7c" exitCode=0 Nov 03 17:00:55.710568 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:55.710244 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" event={"ID":"47230282-26de-49a2-83b0-29a86b164f20","Type":"ContainerDied","Data":"5a9a1cbd38efa5efdf9300246f1ffe4152741ba55922fc7adbc2fe82ad0dde7c"} Nov 03 17:00:55.710568 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:55.710299 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" event={"ID":"47230282-26de-49a2-83b0-29a86b164f20","Type":"ContainerStarted","Data":"90cdf64319f8269cda43e1184abd60ce3e4e6f1d23e842bc0537f8d5de9ceae5"} Nov 03 17:00:56.072083 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.072007 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:56.286635 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.286601 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:56.530622 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.530587 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697"] Nov 03 17:00:56.530863 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.530820 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" containerID="cri-o://e9f7efc14bc85900a644e8c69038e266922531bf0171ebc6c6265ce3829f0d2f" gracePeriod=30 Nov 03 17:00:56.599477 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.599440 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:56.862488 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.862463 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:56.956405 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.956372 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47230282-26de-49a2-83b0-29a86b164f20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47230282-26de-49a2-83b0-29a86b164f20-maas-api-openshift-ca\") pod \"47230282-26de-49a2-83b0-29a86b164f20\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " Nov 03 17:00:56.956561 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.956430 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47230282-26de-49a2-83b0-29a86b164f20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47230282-26de-49a2-83b0-29a86b164f20-maas-api-xks-ca\") pod \"47230282-26de-49a2-83b0-29a86b164f20\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " Nov 03 17:00:56.956561 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.956484 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47230282-26de-49a2-83b0-29a86b164f20-kube-api-access-4sbqv\" (UniqueName: \"kubernetes.io/projected/47230282-26de-49a2-83b0-29a86b164f20-kube-api-access-4sbqv\") pod \"47230282-26de-49a2-83b0-29a86b164f20\" (UID: \"47230282-26de-49a2-83b0-29a86b164f20\") " Nov 03 17:00:56.956742 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.956720 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47230282-26de-49a2-83b0-29a86b164f20-maas-api-openshift-ca" pod "47230282-26de-49a2-83b0-29a86b164f20" (UID: "47230282-26de-49a2-83b0-29a86b164f20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:56.956987 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.956956 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47230282-26de-49a2-83b0-29a86b164f20-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:56.958637 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.958615 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47230282-26de-49a2-83b0-29a86b164f20-kube-api-access-4sbqv" pod "47230282-26de-49a2-83b0-29a86b164f20" (UID: "47230282-26de-49a2-83b0-29a86b164f20"). InnerVolumeSpecName "kube-api-access-4sbqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:56.958753 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:56.958652 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47230282-26de-49a2-83b0-29a86b164f20-maas-api-xks-ca" pod "47230282-26de-49a2-83b0-29a86b164f20" (UID: "47230282-26de-49a2-83b0-29a86b164f20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:57.017462 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.017423 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:57.058035 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.057997 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47230282-26de-49a2-83b0-29a86b164f20-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:57.058035 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.058033 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sbqv\" (UniqueName: \"kubernetes.io/projected/47230282-26de-49a2-83b0-29a86b164f20-kube-api-access-4sbqv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:00:57.220340 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.220251 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:57.513769 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.513740 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh"] Nov 03 17:00:57.514182 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514170 5946 cpu_manager.go:414] "containerMap: removing container" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:57.514245 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514184 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514683 5946 cpu_manager.go:414] "containerMap: removing container" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514738 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42492a4a-b6b9-4f1e-91d1-dd508fa57852" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514816 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514828 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d396844f-cfd1-485c-a52f-33f6d944acd1" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514927 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514953 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc28cee-8a22-4f6d-9f6a-48181d54e3ba" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514988 5946 cpu_manager.go:414] "containerMap: removing container" podUID="47230282-26de-49a2-83b0-29a86b164f20" containerName="revoke-keys" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.514996 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47230282-26de-49a2-83b0-29a86b164f20" containerName="revoke-keys" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.515437 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bdb396-45dd-42ab-be4b-fb8cc7f946d4" containerName="cleanup" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.515458 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="47230282-26de-49a2-83b0-29a86b164f20" containerName="revoke-keys" Nov 03 17:00:57.518237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.515487 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="19311f79-4585-4ddf-894c-a3ba0577d9bf" containerName="cleanup" Nov 03 17:00:57.526815 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.526789 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.563729 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.563697 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.563874 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.563758 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr72p\" (UniqueName: \"kubernetes.io/projected/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-kube-api-access-fr72p\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.563956 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.563935 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.573882 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.573850 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh"] Nov 03 17:00:57.664894 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.664863 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.665093 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.664914 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr72p\" (UniqueName: \"kubernetes.io/projected/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-kube-api-access-fr72p\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.665093 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.664995 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.665711 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.665632 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.667420 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.667398 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.724418 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.724387 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" Nov 03 17:00:57.724418 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.724404 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm" event={"ID":"47230282-26de-49a2-83b0-29a86b164f20","Type":"ContainerDied","Data":"90cdf64319f8269cda43e1184abd60ce3e4e6f1d23e842bc0537f8d5de9ceae5"} Nov 03 17:00:57.724611 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.724467 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90cdf64319f8269cda43e1184abd60ce3e4e6f1d23e842bc0537f8d5de9ceae5" Nov 03 17:00:57.725785 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.725766 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr72p\" (UniqueName: \"kubernetes.io/projected/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-kube-api-access-fr72p\") pod \"maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:57.745358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.745315 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:57.842511 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:57.842428 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:00:58.224744 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:58.224720 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh"] Nov 03 17:00:58.226884 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:00:58.226858 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac27d365_0edb_414a_bcc5_e6b2fa6fade1.slice/crio-252d5bd8bf5dd7dd821741e6f63023d57e7bf85616150b89325d4b98647c1ca7 WatchSource:0}: Error finding container 252d5bd8bf5dd7dd821741e6f63023d57e7bf85616150b89325d4b98647c1ca7: Status 404 returned error can't find the container with id 252d5bd8bf5dd7dd821741e6f63023d57e7bf85616150b89325d4b98647c1ca7 Nov 03 17:00:58.732673 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:58.732636 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" event={"ID":"ac27d365-0edb-414a-bcc5-e6b2fa6fade1","Type":"ContainerStarted","Data":"eff34a5b02722916b94ee498573c46eb666d4795795da082eaafcaa30ba410d6"} Nov 03 17:00:58.732873 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:58.732687 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" event={"ID":"ac27d365-0edb-414a-bcc5-e6b2fa6fade1","Type":"ContainerStarted","Data":"252d5bd8bf5dd7dd821741e6f63023d57e7bf85616150b89325d4b98647c1ca7"} Nov 03 17:00:58.904720 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:58.904611 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" podStartSLOduration=1.904597621 podStartE2EDuration="1.904597621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:58.851503031 +0000 UTC m=+2928.972555855" watchObservedRunningTime="2026-11-03 17:00:58.904597621 +0000 UTC m=+2929.025650438" Nov 03 17:00:59.741313 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:59.741275 5946 generic.go:356] "Generic (PLEG): container finished" podID="ac27d365-0edb-414a-bcc5-e6b2fa6fade1" containerID="eff34a5b02722916b94ee498573c46eb666d4795795da082eaafcaa30ba410d6" exitCode=0 Nov 03 17:00:59.741724 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:00:59.741362 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" event={"ID":"ac27d365-0edb-414a-bcc5-e6b2fa6fade1","Type":"ContainerDied","Data":"eff34a5b02722916b94ee498573c46eb666d4795795da082eaafcaa30ba410d6"} Nov 03 17:01:00.889520 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:00.889486 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:01:00.998831 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:00.998734 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-xks-ca\") pod \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " Nov 03 17:01:00.998998 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:00.998887 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-openshift-ca\") pod \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " Nov 03 17:01:00.998998 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:00.998989 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-kube-api-access-fr72p\" (UniqueName: \"kubernetes.io/projected/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-kube-api-access-fr72p\") pod \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\" (UID: \"ac27d365-0edb-414a-bcc5-e6b2fa6fade1\") " Nov 03 17:01:00.999284 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:00.999260 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-openshift-ca" pod "ac27d365-0edb-414a-bcc5-e6b2fa6fade1" (UID: "ac27d365-0edb-414a-bcc5-e6b2fa6fade1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:01.001120 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.001091 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-xks-ca" pod "ac27d365-0edb-414a-bcc5-e6b2fa6fade1" (UID: "ac27d365-0edb-414a-bcc5-e6b2fa6fade1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:01.001287 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.001140 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-kube-api-access-fr72p" pod "ac27d365-0edb-414a-bcc5-e6b2fa6fade1" (UID: "ac27d365-0edb-414a-bcc5-e6b2fa6fade1"). InnerVolumeSpecName "kube-api-access-fr72p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:01.100355 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.100315 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.100355 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.100350 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.100578 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.100365 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr72p\" (UniqueName: \"kubernetes.io/projected/ac27d365-0edb-414a-bcc5-e6b2fa6fade1-kube-api-access-fr72p\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.511461 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.511422 5946 patch_prober.go:29] interesting pod/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:01:01.511640 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.511479 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:01:01.759003 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.758953 5946 generic.go:356] "Generic (PLEG): container finished" podID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerID="e9f7efc14bc85900a644e8c69038e266922531bf0171ebc6c6265ce3829f0d2f" exitCode=0 Nov 03 17:01:01.759263 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.759028 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" event={"ID":"115e1a7e-4d10-47f8-b610-16383edd7a78","Type":"ContainerDied","Data":"e9f7efc14bc85900a644e8c69038e266922531bf0171ebc6c6265ce3829f0d2f"} Nov 03 17:01:01.759263 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.759102 5946 scope.go:122] "RemoveContainer" containerID="60e54da4d0a4b604f41f9d10f08769cb2bd02c275a6cca6545d9bc65753b9b02" Nov 03 17:01:01.760709 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.760692 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" Nov 03 17:01:01.760789 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.760719 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh" event={"ID":"ac27d365-0edb-414a-bcc5-e6b2fa6fade1","Type":"ContainerDied","Data":"252d5bd8bf5dd7dd821741e6f63023d57e7bf85616150b89325d4b98647c1ca7"} Nov 03 17:01:01.760789 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.760765 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="252d5bd8bf5dd7dd821741e6f63023d57e7bf85616150b89325d4b98647c1ca7" Nov 03 17:01:01.793859 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.793837 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 17:01:01.911400 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911372 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-certs\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911408 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-credential-socket\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911429 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-data\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911453 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-podinfo\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911495 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-token\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-token\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911584 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-kube-api-access-4v9lm\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-kube-api-access-4v9lm\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911693 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115e1a7e-4d10-47f8-b610-16383edd7a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115e1a7e-4d10-47f8-b610-16383edd7a78-istiod-ca-cert\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911688 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-credential-socket" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911716 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-data" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911749 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-envoy\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911826 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-socket\") pod \"115e1a7e-4d10-47f8-b610-16383edd7a78\" (UID: \"115e1a7e-4d10-47f8-b610-16383edd7a78\") " Nov 03 17:01:01.911876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.911769 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-certs" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:01.912409 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.912065 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115e1a7e-4d10-47f8-b610-16383edd7a78-istiod-ca-cert" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:01.912409 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.912153 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-socket" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:01.912625 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.912606 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.912667 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.912635 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.912667 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.912651 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.912740 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.912666 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115e1a7e-4d10-47f8-b610-16383edd7a78-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.912740 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.912682 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:01.913944 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.913916 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-token" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:01.914104 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.913952 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-podinfo" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:01:01.914481 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.914462 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-envoy" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:01.914566 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:01.914537 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-kube-api-access-4v9lm" pod "115e1a7e-4d10-47f8-b610-16383edd7a78" (UID: "115e1a7e-4d10-47f8-b610-16383edd7a78"). InnerVolumeSpecName "kube-api-access-4v9lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:02.013633 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.013597 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9lm\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-kube-api-access-4v9lm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:02.013633 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.013634 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:02.013805 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.013648 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:02.013805 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.013663 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115e1a7e-4d10-47f8-b610-16383edd7a78-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:02.014965 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.014941 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5"] Nov 03 17:01:02.036165 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.036072 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2"] Nov 03 17:01:02.178319 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.178280 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:02.669524 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.669483 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:02.769686 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.769656 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" Nov 03 17:01:02.769686 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.769663 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697" event={"ID":"115e1a7e-4d10-47f8-b610-16383edd7a78","Type":"ContainerDied","Data":"13860c844cbd0ef6def24fc92c22db15b4b7a277e550d5cb20035e8de9461b56"} Nov 03 17:01:02.769919 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.769750 5946 scope.go:122] "RemoveContainer" containerID="e9f7efc14bc85900a644e8c69038e266922531bf0171ebc6c6265ce3829f0d2f" Nov 03 17:01:02.947886 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.947803 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697"] Nov 03 17:01:02.997643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:02.997610 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-61be78-openshift-default-7488fd67c9-5j697"] Nov 03 17:01:04.004954 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:04.004914 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" path="/var/lib/kubelet/pods/115e1a7e-4d10-47f8-b610-16383edd7a78/volumes" Nov 03 17:01:04.914913 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:04.914870 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:05.107304 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:05.107271 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:05.252811 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:05.252776 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9"] Nov 03 17:01:05.252998 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:05.252982 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" containerID="cri-o://4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4" gracePeriod=30 Nov 03 17:01:05.314871 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:05.314790 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:06.053016 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.051680 5946 patch_prober.go:29] interesting pod/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:01:06.053016 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.051757 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:01:06.073920 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.073886 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:06.132883 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.132850 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm"] Nov 03 17:01:06.133557 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.133536 5946 cpu_manager.go:414] "containerMap: removing container" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" Nov 03 17:01:06.133655 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.133558 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" Nov 03 17:01:06.133655 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.133616 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ac27d365-0edb-414a-bcc5-e6b2fa6fade1" containerName="revoke-keys" Nov 03 17:01:06.133655 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.133624 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac27d365-0edb-414a-bcc5-e6b2fa6fade1" containerName="revoke-keys" Nov 03 17:01:06.133828 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.133788 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac27d365-0edb-414a-bcc5-e6b2fa6fade1" containerName="revoke-keys" Nov 03 17:01:06.133828 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.133806 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" Nov 03 17:01:06.133828 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.133820 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" Nov 03 17:01:06.137258 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.137237 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.192787 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.192752 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm"] Nov 03 17:01:06.193328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.193297 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:01:06.193511 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.193395 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:01:06.193511 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.193401 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:01:06.216459 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.216433 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:01:06.261280 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.261243 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.261448 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.261360 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.261448 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.261414 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k449\" (UniqueName: \"kubernetes.io/projected/c93b183d-b5f2-4b50-9533-5255ef217b41-kube-api-access-8k449\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.269329 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.269293 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:06.362981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.362901 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k449\" (UniqueName: \"kubernetes.io/projected/c93b183d-b5f2-4b50-9533-5255ef217b41-kube-api-access-8k449\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.363159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.362984 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.363159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.363035 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.363704 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.363678 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.365600 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.365583 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.423985 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.423952 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k449\" (UniqueName: \"kubernetes.io/projected/c93b183d-b5f2-4b50-9533-5255ef217b41-kube-api-access-8k449\") pod \"maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.450284 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.450254 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:06.603140 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.603106 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:01:06.833000 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:06.832972 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm"] Nov 03 17:01:06.835070 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:01:06.835026 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93b183d_b5f2_4b50_9533_5255ef217b41.slice/crio-7ca0ad4e43d4e774cae98ffc18a97fb6bdc63127440d55545a2022f56a0175d4 WatchSource:0}: Error finding container 7ca0ad4e43d4e774cae98ffc18a97fb6bdc63127440d55545a2022f56a0175d4: Status 404 returned error can't find the container with id 7ca0ad4e43d4e774cae98ffc18a97fb6bdc63127440d55545a2022f56a0175d4 Nov 03 17:01:07.561376 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:07.561343 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:07.805140 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:07.805110 5946 generic.go:356] "Generic (PLEG): container finished" podID="c93b183d-b5f2-4b50-9533-5255ef217b41" containerID="76723dcdbc94e35b706a5551e9c5669d05d05409679b6693cd4a61041541d9cb" exitCode=0 Nov 03 17:01:07.805351 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:07.805221 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" event={"ID":"c93b183d-b5f2-4b50-9533-5255ef217b41","Type":"ContainerDied","Data":"76723dcdbc94e35b706a5551e9c5669d05d05409679b6693cd4a61041541d9cb"} Nov 03 17:01:07.805351 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:07.805277 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" event={"ID":"c93b183d-b5f2-4b50-9533-5255ef217b41","Type":"ContainerStarted","Data":"7ca0ad4e43d4e774cae98ffc18a97fb6bdc63127440d55545a2022f56a0175d4"} Nov 03 17:01:08.961595 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:08.961569 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:09.094583 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.094499 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93b183d-b5f2-4b50-9533-5255ef217b41-kube-api-access-8k449\" (UniqueName: \"kubernetes.io/projected/c93b183d-b5f2-4b50-9533-5255ef217b41-kube-api-access-8k449\") pod \"c93b183d-b5f2-4b50-9533-5255ef217b41\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " Nov 03 17:01:09.094583 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.094557 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-openshift-ca\") pod \"c93b183d-b5f2-4b50-9533-5255ef217b41\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " Nov 03 17:01:09.094845 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.094601 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-xks-ca\") pod \"c93b183d-b5f2-4b50-9533-5255ef217b41\" (UID: \"c93b183d-b5f2-4b50-9533-5255ef217b41\") " Nov 03 17:01:09.094954 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.094928 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-openshift-ca" pod "c93b183d-b5f2-4b50-9533-5255ef217b41" (UID: "c93b183d-b5f2-4b50-9533-5255ef217b41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:09.096944 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.096914 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93b183d-b5f2-4b50-9533-5255ef217b41-kube-api-access-8k449" pod "c93b183d-b5f2-4b50-9533-5255ef217b41" (UID: "c93b183d-b5f2-4b50-9533-5255ef217b41"). InnerVolumeSpecName "kube-api-access-8k449". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:09.097082 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.097059 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-xks-ca" pod "c93b183d-b5f2-4b50-9533-5255ef217b41" (UID: "c93b183d-b5f2-4b50-9533-5255ef217b41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:09.184471 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.184432 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg"] Nov 03 17:01:09.196435 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.196398 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k449\" (UniqueName: \"kubernetes.io/projected/c93b183d-b5f2-4b50-9533-5255ef217b41-kube-api-access-8k449\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:09.196435 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.196430 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:09.196637 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.196443 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93b183d-b5f2-4b50-9533-5255ef217b41-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:09.571403 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.571369 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:09.794962 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.794927 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:09.822305 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.822224 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" Nov 03 17:01:09.822305 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.822251 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm" event={"ID":"c93b183d-b5f2-4b50-9533-5255ef217b41","Type":"ContainerDied","Data":"7ca0ad4e43d4e774cae98ffc18a97fb6bdc63127440d55545a2022f56a0175d4"} Nov 03 17:01:09.822305 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:09.822298 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca0ad4e43d4e774cae98ffc18a97fb6bdc63127440d55545a2022f56a0175d4" Nov 03 17:01:10.508925 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.508898 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 17:01:10.611333 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611244 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-envoy\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611333 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611288 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-kube-api-access-zhnpx\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-kube-api-access-zhnpx\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611555 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611353 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-credential-socket\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611555 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611374 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-certs\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611555 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611445 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-data\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611555 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611463 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istiod-ca-cert\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611555 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611509 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-socket\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611863 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611576 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-podinfo\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611863 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611621 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-token\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-token\") pod \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\" (UID: \"a89e95f7-1357-40ff-b001-cf9878dbdf9b\") " Nov 03 17:01:10.611863 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611697 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-credential-socket" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:10.611863 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611775 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-certs" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:10.611863 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611775 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-data" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:10.612153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611962 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-socket" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:10.612153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.611997 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istiod-ca-cert" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:10.612153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.612102 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.612153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.612122 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.612153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.612133 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.612153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.612141 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.612153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.612153 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.613755 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.613730 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-envoy" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:10.614117 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.614098 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-token" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:10.614246 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.614171 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-kube-api-access-zhnpx" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "kube-api-access-zhnpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:10.614377 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.614360 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-podinfo" pod "a89e95f7-1357-40ff-b001-cf9878dbdf9b" (UID: "a89e95f7-1357-40ff-b001-cf9878dbdf9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:01:10.713436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.713399 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.713436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.713432 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhnpx\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-kube-api-access-zhnpx\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.713436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.713444 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.713711 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.713453 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89e95f7-1357-40ff-b001-cf9878dbdf9b-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:10.831013 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.830976 5946 generic.go:356] "Generic (PLEG): container finished" podID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerID="4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4" exitCode=0 Nov 03 17:01:10.831190 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.831019 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" Nov 03 17:01:10.831190 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.831064 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" event={"ID":"a89e95f7-1357-40ff-b001-cf9878dbdf9b","Type":"ContainerDied","Data":"4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4"} Nov 03 17:01:10.831190 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.831122 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9" event={"ID":"a89e95f7-1357-40ff-b001-cf9878dbdf9b","Type":"ContainerDied","Data":"d5a457ef50a537a329b31f787f5d8f0775f50fa874a93da52a0467af930e1d6d"} Nov 03 17:01:10.831190 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.831135 5946 scope.go:122] "RemoveContainer" containerID="4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4" Nov 03 17:01:10.842127 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.842106 5946 scope.go:122] "RemoveContainer" containerID="f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6" Nov 03 17:01:10.851796 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.851773 5946 scope.go:122] "RemoveContainer" containerID="4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4" Nov 03 17:01:10.852093 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:10.852073 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4\": container with ID starting with 4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4 not found: ID does not exist" containerID="4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4" Nov 03 17:01:10.852139 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.852104 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4"} err="failed to get container status \"4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4\": rpc error: code = NotFound desc = could not find container \"4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4\": container with ID starting with 4c2dd11b276ef1b3dce87bdf193b5c183e8c75addd3ef0d5e60273e1549f39b4 not found: ID does not exist" Nov 03 17:01:10.852139 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.852123 5946 scope.go:122] "RemoveContainer" containerID="f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6" Nov 03 17:01:10.852412 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:10.852394 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6\": container with ID starting with f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6 not found: ID does not exist" containerID="f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6" Nov 03 17:01:10.852459 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:10.852417 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6"} err="failed to get container status \"f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6\": rpc error: code = NotFound desc = could not find container \"f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6\": container with ID starting with f3b72b7d7d4c70d1ec4c9f25830413a9a50aa65ba9237c8f8d5e02bffe8afde6 not found: ID does not exist" Nov 03 17:01:11.002972 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:11.002938 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9"] Nov 03 17:01:11.008743 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:11.008716 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf492-openshift-default-757dff9489-5l7j9"] Nov 03 17:01:12.003799 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.003762 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" path="/var/lib/kubelet/pods/a89e95f7-1357-40ff-b001-cf9878dbdf9b/volumes" Nov 03 17:01:12.018826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.018794 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx"] Nov 03 17:01:12.019284 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019265 5946 cpu_manager.go:414] "containerMap: removing container" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" Nov 03 17:01:12.019284 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019281 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115e1a7e-4d10-47f8-b610-16383edd7a78" containerName="istio-proxy" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019311 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019317 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019329 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c93b183d-b5f2-4b50-9533-5255ef217b41" containerName="revoke-keys" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019334 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93b183d-b5f2-4b50-9533-5255ef217b41" containerName="revoke-keys" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019343 5946 cpu_manager.go:414] "containerMap: removing container" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019347 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019442 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93b183d-b5f2-4b50-9533-5255ef217b41" containerName="revoke-keys" Nov 03 17:01:12.019479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.019457 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" Nov 03 17:01:12.024092 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.024068 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.077174 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.077142 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf"] Nov 03 17:01:12.077749 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.077728 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89e95f7-1357-40ff-b001-cf9878dbdf9b" containerName="istio-proxy" Nov 03 17:01:12.081269 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.081247 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.125712 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.125677 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:01:12.125712 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.125677 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:01:12.128101 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.128079 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx"] Nov 03 17:01:12.128253 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.128233 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.128365 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.128351 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.128408 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.128392 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pspm\" (UniqueName: \"kubernetes.io/projected/b77932d8-246b-45ef-9486-0b1af87117ff-kube-api-access-9pspm\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.129721 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.129701 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:01:12.129782 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.129701 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:01:12.140197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.140171 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf"] Nov 03 17:01:12.229861 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.229814 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.230082 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.229897 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946fd444-b286-4bba-aa60-c758f8789520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.230082 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.229932 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.230082 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.229980 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttpw\" (UniqueName: \"kubernetes.io/projected/946fd444-b286-4bba-aa60-c758f8789520-kube-api-access-tttpw\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.230331 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.230112 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pspm\" (UniqueName: \"kubernetes.io/projected/b77932d8-246b-45ef-9486-0b1af87117ff-kube-api-access-9pspm\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.230331 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.230169 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946fd444-b286-4bba-aa60-c758f8789520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.230655 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.230632 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.232445 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.232422 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.291610 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.291533 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pspm\" (UniqueName: \"kubernetes.io/projected/b77932d8-246b-45ef-9486-0b1af87117ff-kube-api-access-9pspm\") pod \"maas-api-revoke-sub-de0cb1e6-zrjwx\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.331127 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.331084 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946fd444-b286-4bba-aa60-c758f8789520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.331366 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.331162 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttpw\" (UniqueName: \"kubernetes.io/projected/946fd444-b286-4bba-aa60-c758f8789520-kube-api-access-tttpw\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.331366 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.331256 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946fd444-b286-4bba-aa60-c758f8789520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.331880 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.331839 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946fd444-b286-4bba-aa60-c758f8789520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.333794 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.333772 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946fd444-b286-4bba-aa60-c758f8789520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.340793 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.340769 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:12.392970 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.392925 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttpw\" (UniqueName: \"kubernetes.io/projected/946fd444-b286-4bba-aa60-c758f8789520-kube-api-access-tttpw\") pod \"maas-api-revoke-sub-26326065-x4jtf\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.396548 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.396524 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:12.586259 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:12.586148 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:01:12.721392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.721363 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx"] Nov 03 17:01:12.722879 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:01:12.722847 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77932d8_246b_45ef_9486_0b1af87117ff.slice/crio-6dc984a2beb35d4e820ebe23f812815169fe310288450939e66fa4cd69860cdc WatchSource:0}: Error finding container 6dc984a2beb35d4e820ebe23f812815169fe310288450939e66fa4cd69860cdc: Status 404 returned error can't find the container with id 6dc984a2beb35d4e820ebe23f812815169fe310288450939e66fa4cd69860cdc Nov 03 17:01:12.824969 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.824941 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf"] Nov 03 17:01:12.826697 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:01:12.826665 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946fd444_b286_4bba_aa60_c758f8789520.slice/crio-a45373b2f4fbda739d83fe54c17f874b9d1eecbc6db1ba4cbf889467457f7600 WatchSource:0}: Error finding container a45373b2f4fbda739d83fe54c17f874b9d1eecbc6db1ba4cbf889467457f7600: Status 404 returned error can't find the container with id a45373b2f4fbda739d83fe54c17f874b9d1eecbc6db1ba4cbf889467457f7600 Nov 03 17:01:12.846025 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.845989 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" event={"ID":"b77932d8-246b-45ef-9486-0b1af87117ff","Type":"ContainerStarted","Data":"6dc984a2beb35d4e820ebe23f812815169fe310288450939e66fa4cd69860cdc"} Nov 03 17:01:12.847387 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.847362 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" event={"ID":"946fd444-b286-4bba-aa60-c758f8789520","Type":"ContainerStarted","Data":"a45373b2f4fbda739d83fe54c17f874b9d1eecbc6db1ba4cbf889467457f7600"} Nov 03 17:01:12.883094 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:12.883046 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:13.854279 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:13.854248 5946 generic.go:356] "Generic (PLEG): container finished" podID="b77932d8-246b-45ef-9486-0b1af87117ff" containerID="a45caf01aca113cadbe0202a782cae0a9432463b7c87be2ebb640adf4db09765" exitCode=0 Nov 03 17:01:13.854746 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:13.854327 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" event={"ID":"b77932d8-246b-45ef-9486-0b1af87117ff","Type":"ContainerDied","Data":"a45caf01aca113cadbe0202a782cae0a9432463b7c87be2ebb640adf4db09765"} Nov 03 17:01:13.855972 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:13.855949 5946 generic.go:356] "Generic (PLEG): container finished" podID="946fd444-b286-4bba-aa60-c758f8789520" containerID="056c7be22dad2b1e5564e1f05474c45bcf71eab8ebd622d9478e9647526297cd" exitCode=0 Nov 03 17:01:13.856094 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:13.856006 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" event={"ID":"946fd444-b286-4bba-aa60-c758f8789520","Type":"ContainerDied","Data":"056c7be22dad2b1e5564e1f05474c45bcf71eab8ebd622d9478e9647526297cd"} Nov 03 17:01:14.184770 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.184730 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" podUID="157d399c-c209-41dd-b4e4-b4e3cafdc574" containerName="payload-processing" containerID="cri-o://0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6" gracePeriod=25 Nov 03 17:01:14.448288 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.448216 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 17:01:14.555109 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.555076 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157d399c-c209-41dd-b4e4-b4e3cafdc574-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157d399c-c209-41dd-b4e4-b4e3cafdc574-plugins-config-volume\") pod \"157d399c-c209-41dd-b4e4-b4e3cafdc574\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " Nov 03 17:01:14.555369 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.555292 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157d399c-c209-41dd-b4e4-b4e3cafdc574-kube-api-access-vbd6r\" (UniqueName: \"kubernetes.io/projected/157d399c-c209-41dd-b4e4-b4e3cafdc574-kube-api-access-vbd6r\") pod \"157d399c-c209-41dd-b4e4-b4e3cafdc574\" (UID: \"157d399c-c209-41dd-b4e4-b4e3cafdc574\") " Nov 03 17:01:14.555579 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.555553 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157d399c-c209-41dd-b4e4-b4e3cafdc574-plugins-config-volume" pod "157d399c-c209-41dd-b4e4-b4e3cafdc574" (UID: "157d399c-c209-41dd-b4e4-b4e3cafdc574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:14.557619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.557593 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157d399c-c209-41dd-b4e4-b4e3cafdc574-kube-api-access-vbd6r" pod "157d399c-c209-41dd-b4e4-b4e3cafdc574" (UID: "157d399c-c209-41dd-b4e4-b4e3cafdc574"). InnerVolumeSpecName "kube-api-access-vbd6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:14.656877 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.656835 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbd6r\" (UniqueName: \"kubernetes.io/projected/157d399c-c209-41dd-b4e4-b4e3cafdc574-kube-api-access-vbd6r\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:14.656877 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.656869 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157d399c-c209-41dd-b4e4-b4e3cafdc574-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:14.864411 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.864377 5946 generic.go:356] "Generic (PLEG): container finished" podID="157d399c-c209-41dd-b4e4-b4e3cafdc574" containerID="0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6" exitCode=0 Nov 03 17:01:14.864895 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.864425 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" Nov 03 17:01:14.864895 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.864466 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" event={"ID":"157d399c-c209-41dd-b4e4-b4e3cafdc574","Type":"ContainerDied","Data":"0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6"} Nov 03 17:01:14.864895 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.864529 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg" event={"ID":"157d399c-c209-41dd-b4e4-b4e3cafdc574","Type":"ContainerDied","Data":"2b40acd64f96b7416a306b16a1597dd8901ec7c12ee75bbed91be89551250a81"} Nov 03 17:01:14.864895 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.864618 5946 scope.go:122] "RemoveContainer" containerID="0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6" Nov 03 17:01:14.876685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.876655 5946 scope.go:122] "RemoveContainer" containerID="0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6" Nov 03 17:01:14.877002 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:14.876982 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6\": container with ID starting with 0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6 not found: ID does not exist" containerID="0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6" Nov 03 17:01:14.877086 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:14.877010 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6"} err="failed to get container status \"0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6\": rpc error: code = NotFound desc = could not find container \"0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6\": container with ID starting with 0a4f5e796c5ef316c9ef76c5d2a65573905f9be5c0022dad5fb3d6bed6d107b6 not found: ID does not exist" Nov 03 17:01:15.028845 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.028817 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:15.031475 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.031452 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:15.043842 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.043811 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg"] Nov 03 17:01:15.094524 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.094492 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bcd756-6b976556f5-vs7fg"] Nov 03 17:01:15.162849 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.162767 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946fd444-b286-4bba-aa60-c758f8789520-kube-api-access-tttpw\" (UniqueName: \"kubernetes.io/projected/946fd444-b286-4bba-aa60-c758f8789520-kube-api-access-tttpw\") pod \"946fd444-b286-4bba-aa60-c758f8789520\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " Nov 03 17:01:15.162849 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.162836 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946fd444-b286-4bba-aa60-c758f8789520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946fd444-b286-4bba-aa60-c758f8789520-maas-api-xks-ca\") pod \"946fd444-b286-4bba-aa60-c758f8789520\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " Nov 03 17:01:15.163094 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.162862 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-xks-ca\") pod \"b77932d8-246b-45ef-9486-0b1af87117ff\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " Nov 03 17:01:15.163094 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.162885 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-openshift-ca\") pod \"b77932d8-246b-45ef-9486-0b1af87117ff\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " Nov 03 17:01:15.163094 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.162909 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77932d8-246b-45ef-9486-0b1af87117ff-kube-api-access-9pspm\" (UniqueName: \"kubernetes.io/projected/b77932d8-246b-45ef-9486-0b1af87117ff-kube-api-access-9pspm\") pod \"b77932d8-246b-45ef-9486-0b1af87117ff\" (UID: \"b77932d8-246b-45ef-9486-0b1af87117ff\") " Nov 03 17:01:15.163094 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.162935 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946fd444-b286-4bba-aa60-c758f8789520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946fd444-b286-4bba-aa60-c758f8789520-maas-api-openshift-ca\") pod \"946fd444-b286-4bba-aa60-c758f8789520\" (UID: \"946fd444-b286-4bba-aa60-c758f8789520\") " Nov 03 17:01:15.163379 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.163324 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-openshift-ca" pod "b77932d8-246b-45ef-9486-0b1af87117ff" (UID: "b77932d8-246b-45ef-9486-0b1af87117ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:15.163632 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.163600 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946fd444-b286-4bba-aa60-c758f8789520-maas-api-openshift-ca" pod "946fd444-b286-4bba-aa60-c758f8789520" (UID: "946fd444-b286-4bba-aa60-c758f8789520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:15.165138 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.165118 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-xks-ca" pod "b77932d8-246b-45ef-9486-0b1af87117ff" (UID: "b77932d8-246b-45ef-9486-0b1af87117ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:15.165233 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.165138 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946fd444-b286-4bba-aa60-c758f8789520-kube-api-access-tttpw" pod "946fd444-b286-4bba-aa60-c758f8789520" (UID: "946fd444-b286-4bba-aa60-c758f8789520"). InnerVolumeSpecName "kube-api-access-tttpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:15.165522 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.165502 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946fd444-b286-4bba-aa60-c758f8789520-maas-api-xks-ca" pod "946fd444-b286-4bba-aa60-c758f8789520" (UID: "946fd444-b286-4bba-aa60-c758f8789520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:15.165589 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.165562 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77932d8-246b-45ef-9486-0b1af87117ff-kube-api-access-9pspm" pod "b77932d8-246b-45ef-9486-0b1af87117ff" (UID: "b77932d8-246b-45ef-9486-0b1af87117ff"). InnerVolumeSpecName "kube-api-access-9pspm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:15.264113 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.264070 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tttpw\" (UniqueName: \"kubernetes.io/projected/946fd444-b286-4bba-aa60-c758f8789520-kube-api-access-tttpw\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:15.264113 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.264107 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946fd444-b286-4bba-aa60-c758f8789520-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:15.264113 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.264120 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:15.264403 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.264134 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b77932d8-246b-45ef-9486-0b1af87117ff-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:15.264403 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.264150 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pspm\" (UniqueName: \"kubernetes.io/projected/b77932d8-246b-45ef-9486-0b1af87117ff-kube-api-access-9pspm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:15.264403 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.264162 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946fd444-b286-4bba-aa60-c758f8789520-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:15.872951 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.872921 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" Nov 03 17:01:15.873485 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.872956 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf" event={"ID":"946fd444-b286-4bba-aa60-c758f8789520","Type":"ContainerDied","Data":"a45373b2f4fbda739d83fe54c17f874b9d1eecbc6db1ba4cbf889467457f7600"} Nov 03 17:01:15.873485 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.873017 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45373b2f4fbda739d83fe54c17f874b9d1eecbc6db1ba4cbf889467457f7600" Nov 03 17:01:15.875085 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.875062 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" event={"ID":"b77932d8-246b-45ef-9486-0b1af87117ff","Type":"ContainerDied","Data":"6dc984a2beb35d4e820ebe23f812815169fe310288450939e66fa4cd69860cdc"} Nov 03 17:01:15.875197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.875087 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx" Nov 03 17:01:15.875197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:15.875094 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dc984a2beb35d4e820ebe23f812815169fe310288450939e66fa4cd69860cdc" Nov 03 17:01:16.003297 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:16.003264 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157d399c-c209-41dd-b4e4-b4e3cafdc574" path="/var/lib/kubelet/pods/157d399c-c209-41dd-b4e4-b4e3cafdc574/volumes" Nov 03 17:01:17.038182 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.038135 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" containerID="cri-o://7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643" gracePeriod=45 Nov 03 17:01:17.064691 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.064656 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" containerID="cri-o://1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524" gracePeriod=45 Nov 03 17:01:17.334866 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.334840 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 17:01:17.362833 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.362804 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486624 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-dshm\") pod \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486731 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-model-cache\") pod \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486764 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-model-cache\") pod \"7cccbb33-75ea-43de-9340-41f1a7e45811\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486793 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-kube-api-access-mdxm2\" (UniqueName: \"kubernetes.io/projected/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-kube-api-access-mdxm2\") pod \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486871 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cccbb33-75ea-43de-9340-41f1a7e45811-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cccbb33-75ea-43de-9340-41f1a7e45811-tls-certs\") pod \"7cccbb33-75ea-43de-9340-41f1a7e45811\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486907 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-home\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-home\") pod \"7cccbb33-75ea-43de-9340-41f1a7e45811\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486944 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cccbb33-75ea-43de-9340-41f1a7e45811-kube-api-access-f4rls\" (UniqueName: \"kubernetes.io/projected/7cccbb33-75ea-43de-9340-41f1a7e45811-kube-api-access-f4rls\") pod \"7cccbb33-75ea-43de-9340-41f1a7e45811\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.486979 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-home\") pod \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.487018 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-tmp-dir\") pod \"7cccbb33-75ea-43de-9340-41f1a7e45811\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.487054 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tls-certs\") pod \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.487082 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tmp-dir\") pod \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\" (UID: \"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da\") " Nov 03 17:01:17.487230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.487106 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-dshm\") pod \"7cccbb33-75ea-43de-9340-41f1a7e45811\" (UID: \"7cccbb33-75ea-43de-9340-41f1a7e45811\") " Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.488065 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-home" pod "7cccbb33-75ea-43de-9340-41f1a7e45811" (UID: "7cccbb33-75ea-43de-9340-41f1a7e45811"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.490117 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-model-cache" pod "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" (UID: "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.490341 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-tmp-dir" pod "7cccbb33-75ea-43de-9340-41f1a7e45811" (UID: "7cccbb33-75ea-43de-9340-41f1a7e45811"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.491286 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-home" pod "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" (UID: "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.491480 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tmp-dir" pod "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" (UID: "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.491711 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-model-cache" pod "7cccbb33-75ea-43de-9340-41f1a7e45811" (UID: "7cccbb33-75ea-43de-9340-41f1a7e45811"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.494519 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cccbb33-75ea-43de-9340-41f1a7e45811-kube-api-access-f4rls" pod "7cccbb33-75ea-43de-9340-41f1a7e45811" (UID: "7cccbb33-75ea-43de-9340-41f1a7e45811"). InnerVolumeSpecName "kube-api-access-f4rls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.494819 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-dshm" pod "7cccbb33-75ea-43de-9340-41f1a7e45811" (UID: "7cccbb33-75ea-43de-9340-41f1a7e45811"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.494874 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-dshm" pod "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" (UID: "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.496485 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-kube-api-access-mdxm2" pod "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" (UID: "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da"). InnerVolumeSpecName "kube-api-access-mdxm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.496582 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tls-certs" pod "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" (UID: "9fc48d03-da3f-4e61-9cf3-fadfaf0c27da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:17.498106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.497955 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cccbb33-75ea-43de-9340-41f1a7e45811-tls-certs" pod "7cccbb33-75ea-43de-9340-41f1a7e45811" (UID: "7cccbb33-75ea-43de-9340-41f1a7e45811"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589027 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589062 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589073 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdxm2\" (UniqueName: \"kubernetes.io/projected/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-kube-api-access-mdxm2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589085 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cccbb33-75ea-43de-9340-41f1a7e45811-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589093 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589104 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rls\" (UniqueName: \"kubernetes.io/projected/7cccbb33-75ea-43de-9340-41f1a7e45811-kube-api-access-f4rls\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589112 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589122 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589134 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589142 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589150 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cccbb33-75ea-43de-9340-41f1a7e45811-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.589254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.589156 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:17.682603 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.682565 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:17.723801 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.723765 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7"] Nov 03 17:01:17.888929 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.888841 5946 generic.go:356] "Generic (PLEG): container finished" podID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerID="7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643" exitCode=0 Nov 03 17:01:17.888929 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.888895 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" Nov 03 17:01:17.888929 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.888902 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" event={"ID":"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da","Type":"ContainerDied","Data":"7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643"} Nov 03 17:01:17.889180 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.888938 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5" event={"ID":"9fc48d03-da3f-4e61-9cf3-fadfaf0c27da","Type":"ContainerDied","Data":"f7f7d86540d0e786c992e90c0133479a0caaf9a2ec2d831f0587245b0fd01271"} Nov 03 17:01:17.889180 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.888955 5946 scope.go:122] "RemoveContainer" containerID="7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643" Nov 03 17:01:17.890448 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.890420 5946 generic.go:356] "Generic (PLEG): container finished" podID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerID="1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524" exitCode=0 Nov 03 17:01:17.890619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.890491 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" Nov 03 17:01:17.890619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.890519 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" event={"ID":"7cccbb33-75ea-43de-9340-41f1a7e45811","Type":"ContainerDied","Data":"1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524"} Nov 03 17:01:17.890619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.890580 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2" event={"ID":"7cccbb33-75ea-43de-9340-41f1a7e45811","Type":"ContainerDied","Data":"bd673f4fe30b9212136b8211bbbd656ffc3d0b920c6d7c63f28979c095295384"} Nov 03 17:01:17.901742 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.901719 5946 scope.go:122] "RemoveContainer" containerID="7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643" Nov 03 17:01:17.902062 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:17.902042 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643\": container with ID starting with 7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643 not found: ID does not exist" containerID="7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643" Nov 03 17:01:17.902160 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.902076 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643"} err="failed to get container status \"7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643\": rpc error: code = NotFound desc = could not find container \"7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643\": container with ID starting with 7fe57c1c38305951f23a06408c2fedd9116fea4d13e70aa667aea90a7df2d643 not found: ID does not exist" Nov 03 17:01:17.902160 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.902100 5946 scope.go:122] "RemoveContainer" containerID="1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524" Nov 03 17:01:17.914770 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.914747 5946 scope.go:122] "RemoveContainer" containerID="1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524" Nov 03 17:01:17.915100 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:17.915080 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524\": container with ID starting with 1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524 not found: ID does not exist" containerID="1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524" Nov 03 17:01:17.915173 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:17.915109 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524"} err="failed to get container status \"1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524\": rpc error: code = NotFound desc = could not find container \"1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524\": container with ID starting with 1b8f38e46fc716e8c97436a8bc39e6c559e8b00a65b6c7cacb48b91b28550524 not found: ID does not exist" Nov 03 17:01:18.002593 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:18.002563 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:18.060922 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:18.060893 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5"] Nov 03 17:01:18.068808 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:18.068780 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/distinct-w1-bcd756-kserve-5fc9897569-k8lt5"] Nov 03 17:01:18.182416 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:18.182320 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2"] Nov 03 17:01:18.188573 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:18.188534 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:18.190594 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:18.190572 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/simulator-w1-bcd756-kserve-f56ddcc75-hspg2"] Nov 03 17:01:19.423989 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:19.423957 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2"] Nov 03 17:01:19.424442 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:19.424178 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" containerID="cri-o://b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2" gracePeriod=30 Nov 03 17:01:20.002855 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:20.002815 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" path="/var/lib/kubelet/pods/7cccbb33-75ea-43de-9340-41f1a7e45811/volumes" Nov 03 17:01:20.003315 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:20.003293 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" path="/var/lib/kubelet/pods/9fc48d03-da3f-4e61-9cf3-fadfaf0c27da/volumes" Nov 03 17:01:21.836873 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:21.836831 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:22.724136 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:22.724092 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" podUID="1b0ace67-04fd-41f6-898b-e00a64411257" containerName="payload-pre-processing" containerID="cri-o://61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55" gracePeriod=25 Nov 03 17:01:23.352406 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:23.352369 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:23.584150 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:23.584111 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:23.799275 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:23.799238 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:23.864784 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:23.864746 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm"] Nov 03 17:01:23.915003 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:23.914968 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-195095-zl6nm"] Nov 03 17:01:24.003743 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.003714 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47230282-26de-49a2-83b0-29a86b164f20" path="/var/lib/kubelet/pods/47230282-26de-49a2-83b0-29a86b164f20/volumes" Nov 03 17:01:24.337897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.337861 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:24.674329 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.674304 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 17:01:24.867685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867613 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-envoy\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.867685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867663 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac72db61-3320-4470-b8cd-872c9d48b4f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72db61-3320-4470-b8cd-872c9d48b4f4-istiod-ca-cert\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.867896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867723 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-certs\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.867896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867775 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-kube-api-access-lch2d\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-kube-api-access-lch2d\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.867896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867797 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-data\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.867896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867836 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-credential-socket\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.867896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867862 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-socket\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.868164 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867930 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-podinfo\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.868164 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.867980 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-token\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-token\") pod \"ac72db61-3320-4470-b8cd-872c9d48b4f4\" (UID: \"ac72db61-3320-4470-b8cd-872c9d48b4f4\") " Nov 03 17:01:24.868164 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868132 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-credential-socket" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:24.868164 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868148 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-data" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:24.868396 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868191 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac72db61-3320-4470-b8cd-872c9d48b4f4-istiod-ca-cert" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:24.868567 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868485 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.868567 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868511 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.868567 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868526 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72db61-3320-4470-b8cd-872c9d48b4f4-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.868821 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868586 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-socket" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:24.868821 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.868771 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-certs" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:24.870089 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.870067 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-envoy" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:24.870398 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.870370 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-podinfo" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:01:24.870674 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.870660 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-kube-api-access-lch2d" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "kube-api-access-lch2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:24.870728 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.870682 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-token" pod "ac72db61-3320-4470-b8cd-872c9d48b4f4" (UID: "ac72db61-3320-4470-b8cd-872c9d48b4f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:24.938693 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.938658 5946 generic.go:356] "Generic (PLEG): container finished" podID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerID="b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2" exitCode=0 Nov 03 17:01:24.938844 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.938695 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" Nov 03 17:01:24.938844 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.938745 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" event={"ID":"ac72db61-3320-4470-b8cd-872c9d48b4f4","Type":"ContainerDied","Data":"b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2"} Nov 03 17:01:24.938844 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.938799 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2" event={"ID":"ac72db61-3320-4470-b8cd-872c9d48b4f4","Type":"ContainerDied","Data":"71a3cdf6b93a656b00a324014c307cc88c3b73e6b41433c672cde8af64e9fbc0"} Nov 03 17:01:24.938844 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.938825 5946 scope.go:122] "RemoveContainer" containerID="b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2" Nov 03 17:01:24.949572 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.949548 5946 scope.go:122] "RemoveContainer" containerID="b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2" Nov 03 17:01:24.949835 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:24.949809 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2\": container with ID starting with b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2 not found: ID does not exist" containerID="b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2" Nov 03 17:01:24.949884 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.949851 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2"} err="failed to get container status \"b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2\": rpc error: code = NotFound desc = could not find container \"b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2\": container with ID starting with b1f1097a472e233af8131d42b6d868bd7a9d29716443fffec7a84ecc6c1d51a2 not found: ID does not exist" Nov 03 17:01:24.969518 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.969482 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.969518 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.969514 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.969518 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.969523 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.969728 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.969532 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.969728 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.969541 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lch2d\" (UniqueName: \"kubernetes.io/projected/ac72db61-3320-4470-b8cd-872c9d48b4f4-kube-api-access-lch2d\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:24.969728 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:24.969552 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72db61-3320-4470-b8cd-872c9d48b4f4-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:25.114631 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:25.114599 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2"] Nov 03 17:01:25.121867 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:25.121783 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-j5jx2"] Nov 03 17:01:26.002854 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:26.002819 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" path="/var/lib/kubelet/pods/ac72db61-3320-4470-b8cd-872c9d48b4f4/volumes" Nov 03 17:01:28.057384 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:28.057332 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:29.101482 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:29.101441 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" podUID="1b0ace67-04fd-41f6-898b-e00a64411257" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:01:29.556095 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:29.556055 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:29.753615 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:29.753568 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:29.944053 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:29.943969 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:30.036551 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:30.036514 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm"] Nov 03 17:01:30.081303 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:30.081270 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963a2f-j6fxm"] Nov 03 17:01:30.589762 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:30.589708 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:32.003350 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:32.003317 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93b183d-b5f2-4b50-9533-5255ef217b41" path="/var/lib/kubelet/pods/c93b183d-b5f2-4b50-9533-5255ef217b41/volumes" Nov 03 17:01:32.464339 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:32.464253 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:32.678741 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:32.678694 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:33.307845 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:33.307814 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:34.777417 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:34.777388 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:38.014975 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:38.014926 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:38.318239 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:38.318140 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:38.508816 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:38.508787 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:39.101762 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.101717 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" podUID="1b0ace67-04fd-41f6-898b-e00a64411257" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:01:39.500948 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.500901 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:39.585902 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.585867 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c"] Nov 03 17:01:39.586304 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586291 5946 cpu_manager.go:414] "containerMap: removing container" podUID="157d399c-c209-41dd-b4e4-b4e3cafdc574" containerName="payload-processing" Nov 03 17:01:39.586358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586306 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157d399c-c209-41dd-b4e4-b4e3cafdc574" containerName="payload-processing" Nov 03 17:01:39.586358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586325 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" Nov 03 17:01:39.586358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586331 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" Nov 03 17:01:39.586358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586345 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" Nov 03 17:01:39.586358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586349 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" Nov 03 17:01:39.586358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586357 5946 cpu_manager.go:414] "containerMap: removing container" podUID="b77932d8-246b-45ef-9486-0b1af87117ff" containerName="revoke-keys" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586362 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77932d8-246b-45ef-9486-0b1af87117ff" containerName="revoke-keys" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586378 5946 cpu_manager.go:414] "containerMap: removing container" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586383 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586397 5946 cpu_manager.go:414] "containerMap: removing container" podUID="946fd444-b286-4bba-aa60-c758f8789520" containerName="revoke-keys" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586401 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946fd444-b286-4bba-aa60-c758f8789520" containerName="revoke-keys" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586463 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac72db61-3320-4470-b8cd-872c9d48b4f4" containerName="istio-proxy" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586470 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77932d8-246b-45ef-9486-0b1af87117ff" containerName="revoke-keys" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586478 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc48d03-da3f-4e61-9cf3-fadfaf0c27da" containerName="main" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586485 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="946fd444-b286-4bba-aa60-c758f8789520" containerName="revoke-keys" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586491 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="157d399c-c209-41dd-b4e4-b4e3cafdc574" containerName="payload-processing" Nov 03 17:01:39.586565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.586499 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cccbb33-75ea-43de-9340-41f1a7e45811" containerName="main" Nov 03 17:01:39.588849 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.588833 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.682642 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.682584 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:01:39.682856 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.682693 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:01:39.682856 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.682704 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:01:39.684297 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.684274 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:01:39.686252 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.686231 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c"] Nov 03 17:01:39.713844 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.713810 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.713995 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.713875 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.713995 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.713949 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7q2\" (UniqueName: \"kubernetes.io/projected/429d5bb8-e492-451d-b09f-dea7191ed9ea-kube-api-access-jv7q2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.745049 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.744997 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:01:39.814991 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.814913 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.814991 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.814956 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7q2\" (UniqueName: \"kubernetes.io/projected/429d5bb8-e492-451d-b09f-dea7191ed9ea-kube-api-access-jv7q2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.815345 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.815043 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.815782 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.815757 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.817754 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.817731 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.875759 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.875723 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7q2\" (UniqueName: \"kubernetes.io/projected/429d5bb8-e492-451d-b09f-dea7191ed9ea-kube-api-access-jv7q2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:39.901240 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:39.901189 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:40.013920 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:40.013887 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:40.207859 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:40.207779 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:40.298746 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:40.298703 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c"] Nov 03 17:01:40.299281 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:01:40.299254 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429d5bb8_e492_451d_b09f_dea7191ed9ea.slice/crio-b35a47601e9a0b1ec3740aed3a3b52fa2d8fc48ce9101b48b559417f3b0b5a64 WatchSource:0}: Error finding container b35a47601e9a0b1ec3740aed3a3b52fa2d8fc48ce9101b48b559417f3b0b5a64: Status 404 returned error can't find the container with id b35a47601e9a0b1ec3740aed3a3b52fa2d8fc48ce9101b48b559417f3b0b5a64 Nov 03 17:01:41.046122 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:41.046082 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" event={"ID":"429d5bb8-e492-451d-b09f-dea7191ed9ea","Type":"ContainerStarted","Data":"564b19039fccd9156ab4f605bbdd8f59bbe99c6f7e0e4e6add0f4a254c4a1818"} Nov 03 17:01:41.046343 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:41.046183 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" event={"ID":"429d5bb8-e492-451d-b09f-dea7191ed9ea","Type":"ContainerStarted","Data":"b35a47601e9a0b1ec3740aed3a3b52fa2d8fc48ce9101b48b559417f3b0b5a64"} Nov 03 17:01:41.171076 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:41.171009 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" podStartSLOduration=2.170990019 podStartE2EDuration="2.170990019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:41.168332169 +0000 UTC m=+2971.289384994" watchObservedRunningTime="2026-11-03 17:01:41.170990019 +0000 UTC m=+2971.292042845" Nov 03 17:01:41.200486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:41.200453 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:42.053178 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:42.053143 5946 generic.go:356] "Generic (PLEG): container finished" podID="429d5bb8-e492-451d-b09f-dea7191ed9ea" containerID="564b19039fccd9156ab4f605bbdd8f59bbe99c6f7e0e4e6add0f4a254c4a1818" exitCode=0 Nov 03 17:01:42.053572 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:42.053231 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" event={"ID":"429d5bb8-e492-451d-b09f-dea7191ed9ea","Type":"ContainerDied","Data":"564b19039fccd9156ab4f605bbdd8f59bbe99c6f7e0e4e6add0f4a254c4a1818"} Nov 03 17:01:42.585110 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:42.585060 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:01:42.937492 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:42.937460 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:42.977084 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:42.977061 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 17:01:43.047267 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.047229 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0ace67-04fd-41f6-898b-e00a64411257-kube-api-access-7488k\" (UniqueName: \"kubernetes.io/projected/1b0ace67-04fd-41f6-898b-e00a64411257-kube-api-access-7488k\") pod \"1b0ace67-04fd-41f6-898b-e00a64411257\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " Nov 03 17:01:43.047451 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.047317 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0ace67-04fd-41f6-898b-e00a64411257-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0ace67-04fd-41f6-898b-e00a64411257-plugins-config-volume\") pod \"1b0ace67-04fd-41f6-898b-e00a64411257\" (UID: \"1b0ace67-04fd-41f6-898b-e00a64411257\") " Nov 03 17:01:43.047722 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.047696 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0ace67-04fd-41f6-898b-e00a64411257-plugins-config-volume" pod "1b0ace67-04fd-41f6-898b-e00a64411257" (UID: "1b0ace67-04fd-41f6-898b-e00a64411257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:43.049523 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.049486 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0ace67-04fd-41f6-898b-e00a64411257-kube-api-access-7488k" pod "1b0ace67-04fd-41f6-898b-e00a64411257" (UID: "1b0ace67-04fd-41f6-898b-e00a64411257"). InnerVolumeSpecName "kube-api-access-7488k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:43.060780 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.060754 5946 generic.go:356] "Generic (PLEG): container finished" podID="1b0ace67-04fd-41f6-898b-e00a64411257" containerID="61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55" exitCode=0 Nov 03 17:01:43.061201 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.060809 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" Nov 03 17:01:43.061201 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.060828 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" event={"ID":"1b0ace67-04fd-41f6-898b-e00a64411257","Type":"ContainerDied","Data":"61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55"} Nov 03 17:01:43.061201 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.060884 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7" event={"ID":"1b0ace67-04fd-41f6-898b-e00a64411257","Type":"ContainerDied","Data":"0f2e4f61b0e971839ab574900a71d99512e184fe8708e42c255745421af0d122"} Nov 03 17:01:43.061201 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.060906 5946 scope.go:122] "RemoveContainer" containerID="61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55" Nov 03 17:01:43.071851 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.071826 5946 scope.go:122] "RemoveContainer" containerID="61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55" Nov 03 17:01:43.072168 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:43.072149 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55\": container with ID starting with 61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55 not found: ID does not exist" containerID="61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55" Nov 03 17:01:43.072240 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.072178 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55"} err="failed to get container status \"61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55\": rpc error: code = NotFound desc = could not find container \"61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55\": container with ID starting with 61b63e043a07838b63d93f7041b665e22c664fabfcebe9eac99e933105be9f55 not found: ID does not exist" Nov 03 17:01:43.149156 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.149119 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7488k\" (UniqueName: \"kubernetes.io/projected/1b0ace67-04fd-41f6-898b-e00a64411257-kube-api-access-7488k\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:43.149156 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.149157 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0ace67-04fd-41f6-898b-e00a64411257-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:43.213840 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.213812 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7"] Nov 03 17:01:43.219755 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.219733 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:43.219985 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.219966 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963a2f-6454f74599-s4ww7"] Nov 03 17:01:43.351042 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.351005 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429d5bb8-e492-451d-b09f-dea7191ed9ea-kube-api-access-jv7q2\" (UniqueName: \"kubernetes.io/projected/429d5bb8-e492-451d-b09f-dea7191ed9ea-kube-api-access-jv7q2\") pod \"429d5bb8-e492-451d-b09f-dea7191ed9ea\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " Nov 03 17:01:43.351277 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.351107 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-xks-ca\") pod \"429d5bb8-e492-451d-b09f-dea7191ed9ea\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " Nov 03 17:01:43.351277 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.351143 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-openshift-ca\") pod \"429d5bb8-e492-451d-b09f-dea7191ed9ea\" (UID: \"429d5bb8-e492-451d-b09f-dea7191ed9ea\") " Nov 03 17:01:43.351522 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.351485 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-openshift-ca" pod "429d5bb8-e492-451d-b09f-dea7191ed9ea" (UID: "429d5bb8-e492-451d-b09f-dea7191ed9ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:43.351626 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.351607 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:43.353306 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.353281 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429d5bb8-e492-451d-b09f-dea7191ed9ea-kube-api-access-jv7q2" pod "429d5bb8-e492-451d-b09f-dea7191ed9ea" (UID: "429d5bb8-e492-451d-b09f-dea7191ed9ea"). InnerVolumeSpecName "kube-api-access-jv7q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:43.353446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.353323 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-xks-ca" pod "429d5bb8-e492-451d-b09f-dea7191ed9ea" (UID: "429d5bb8-e492-451d-b09f-dea7191ed9ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:43.452827 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.452744 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv7q2\" (UniqueName: \"kubernetes.io/projected/429d5bb8-e492-451d-b09f-dea7191ed9ea-kube-api-access-jv7q2\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:43.452827 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.452777 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d5bb8-e492-451d-b09f-dea7191ed9ea-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:43.913833 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:43.913794 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:44.004445 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.004365 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0ace67-04fd-41f6-898b-e00a64411257" path="/var/lib/kubelet/pods/1b0ace67-04fd-41f6-898b-e00a64411257/volumes" Nov 03 17:01:44.067806 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.067774 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" Nov 03 17:01:44.067806 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.067804 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c" event={"ID":"429d5bb8-e492-451d-b09f-dea7191ed9ea","Type":"ContainerDied","Data":"b35a47601e9a0b1ec3740aed3a3b52fa2d8fc48ce9101b48b559417f3b0b5a64"} Nov 03 17:01:44.068331 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.067861 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b35a47601e9a0b1ec3740aed3a3b52fa2d8fc48ce9101b48b559417f3b0b5a64" Nov 03 17:01:44.106107 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.106074 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:44.611934 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.611897 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9"] Nov 03 17:01:44.612388 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.612371 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1b0ace67-04fd-41f6-898b-e00a64411257" containerName="payload-pre-processing" Nov 03 17:01:44.612449 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.612388 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0ace67-04fd-41f6-898b-e00a64411257" containerName="payload-pre-processing" Nov 03 17:01:44.612449 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.612432 5946 cpu_manager.go:414] "containerMap: removing container" podUID="429d5bb8-e492-451d-b09f-dea7191ed9ea" containerName="revoke-keys" Nov 03 17:01:44.612449 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.612438 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429d5bb8-e492-451d-b09f-dea7191ed9ea" containerName="revoke-keys" Nov 03 17:01:44.612558 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.612504 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="429d5bb8-e492-451d-b09f-dea7191ed9ea" containerName="revoke-keys" Nov 03 17:01:44.612558 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.612514 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0ace67-04fd-41f6-898b-e00a64411257" containerName="payload-pre-processing" Nov 03 17:01:44.615347 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.615328 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.664515 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.664484 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.664685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.664540 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcjv\" (UniqueName: \"kubernetes.io/projected/f910b760-1ce1-4d3b-81a6-6842ce741f3a-kube-api-access-hbcjv\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.664685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.664627 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.673388 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.673359 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:01:44.675745 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.675720 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9"] Nov 03 17:01:44.723179 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.723144 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:01:44.723413 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.723392 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:01:44.723491 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.723395 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:01:44.765910 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.765861 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.766107 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.765928 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcjv\" (UniqueName: \"kubernetes.io/projected/f910b760-1ce1-4d3b-81a6-6842ce741f3a-kube-api-access-hbcjv\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.766107 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.765967 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.766668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.766643 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.768487 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.768464 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.827801 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.827772 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcjv\" (UniqueName: \"kubernetes.io/projected/f910b760-1ce1-4d3b-81a6-6842ce741f3a-kube-api-access-hbcjv\") pod \"maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:44.900317 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.900253 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:44.926948 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:44.926923 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:45.305631 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:45.305600 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9"] Nov 03 17:01:45.307098 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:01:45.307065 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf910b760_1ce1_4d3b_81a6_6842ce741f3a.slice/crio-eb6268e291ba5ebf6bbb37e05c9aeda3db5af2ca04f090f3c8cab6bff1f78307 WatchSource:0}: Error finding container eb6268e291ba5ebf6bbb37e05c9aeda3db5af2ca04f090f3c8cab6bff1f78307: Status 404 returned error can't find the container with id eb6268e291ba5ebf6bbb37e05c9aeda3db5af2ca04f090f3c8cab6bff1f78307 Nov 03 17:01:45.452377 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:45.452344 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:45.763362 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:45.763327 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:45.935595 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:45.935502 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh"] Nov 03 17:01:45.940371 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:45.940344 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bcd756-qwczh"] Nov 03 17:01:45.956954 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:45.956924 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:46.012428 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:46.012388 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac27d365-0edb-414a-bcc5-e6b2fa6fade1" path="/var/lib/kubelet/pods/ac27d365-0edb-414a-bcc5-e6b2fa6fade1/volumes" Nov 03 17:01:46.082381 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:46.082347 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" event={"ID":"f910b760-1ce1-4d3b-81a6-6842ce741f3a","Type":"ContainerStarted","Data":"3e2cc0a2b5b7b53cd0c893bf69bdfc07bea28f4b6ea852dd25cec81ad7a1a7b0"} Nov 03 17:01:46.082535 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:46.082396 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" event={"ID":"f910b760-1ce1-4d3b-81a6-6842ce741f3a","Type":"ContainerStarted","Data":"eb6268e291ba5ebf6bbb37e05c9aeda3db5af2ca04f090f3c8cab6bff1f78307"} Nov 03 17:01:46.206014 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:46.205897 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" podStartSLOduration=2.205883858 podStartE2EDuration="2.205883858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:46.204303036 +0000 UTC m=+2976.325355860" watchObservedRunningTime="2026-11-03 17:01:46.205883858 +0000 UTC m=+2976.326936674" Nov 03 17:01:46.418585 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:46.418543 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:47.090306 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:47.090250 5946 generic.go:356] "Generic (PLEG): container finished" podID="f910b760-1ce1-4d3b-81a6-6842ce741f3a" containerID="3e2cc0a2b5b7b53cd0c893bf69bdfc07bea28f4b6ea852dd25cec81ad7a1a7b0" exitCode=0 Nov 03 17:01:47.090521 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:47.090342 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" event={"ID":"f910b760-1ce1-4d3b-81a6-6842ce741f3a","Type":"ContainerDied","Data":"3e2cc0a2b5b7b53cd0c893bf69bdfc07bea28f4b6ea852dd25cec81ad7a1a7b0"} Nov 03 17:01:48.252949 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.252922 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:48.405939 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.405867 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-xks-ca\") pod \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " Nov 03 17:01:48.406078 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.405978 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-openshift-ca\") pod \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " Nov 03 17:01:48.406078 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.406015 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f910b760-1ce1-4d3b-81a6-6842ce741f3a-kube-api-access-hbcjv\" (UniqueName: \"kubernetes.io/projected/f910b760-1ce1-4d3b-81a6-6842ce741f3a-kube-api-access-hbcjv\") pod \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\" (UID: \"f910b760-1ce1-4d3b-81a6-6842ce741f3a\") " Nov 03 17:01:48.406363 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.406337 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-openshift-ca" pod "f910b760-1ce1-4d3b-81a6-6842ce741f3a" (UID: "f910b760-1ce1-4d3b-81a6-6842ce741f3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:48.408247 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.408194 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-xks-ca" pod "f910b760-1ce1-4d3b-81a6-6842ce741f3a" (UID: "f910b760-1ce1-4d3b-81a6-6842ce741f3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:48.408335 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.408198 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f910b760-1ce1-4d3b-81a6-6842ce741f3a-kube-api-access-hbcjv" pod "f910b760-1ce1-4d3b-81a6-6842ce741f3a" (UID: "f910b760-1ce1-4d3b-81a6-6842ce741f3a"). InnerVolumeSpecName "kube-api-access-hbcjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:48.507166 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.507135 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:48.507166 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.507163 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcjv\" (UniqueName: \"kubernetes.io/projected/f910b760-1ce1-4d3b-81a6-6842ce741f3a-kube-api-access-hbcjv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:48.507166 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.507176 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f910b760-1ce1-4d3b-81a6-6842ce741f3a-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:48.983597 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:48.983566 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6"] Nov 03 17:01:49.034611 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:49.034573 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg"] Nov 03 17:01:49.036624 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:49.036593 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54"] Nov 03 17:01:49.103920 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:49.103887 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" event={"ID":"f910b760-1ce1-4d3b-81a6-6842ce741f3a","Type":"ContainerDied","Data":"eb6268e291ba5ebf6bbb37e05c9aeda3db5af2ca04f090f3c8cab6bff1f78307"} Nov 03 17:01:49.103920 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:49.103916 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9" Nov 03 17:01:49.104140 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:49.103933 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb6268e291ba5ebf6bbb37e05c9aeda3db5af2ca04f090f3c8cab6bff1f78307" Nov 03 17:01:49.179954 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:49.179901 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:50.219217 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:50.219178 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:50.563645 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:50.563604 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks"] Nov 03 17:01:50.615232 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:50.615185 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds"] Nov 03 17:01:50.770333 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:50.770305 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:50.969432 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:50.969318 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:51.157702 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:51.157648 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:51.994532 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:51.994458 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:54.298774 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:54.298732 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:54.509043 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:54.509008 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:54.704950 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:54.704883 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:55.028351 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.028321 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:55.304920 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.304810 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:55.515501 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.515470 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c"] Nov 03 17:01:55.564125 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.564018 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" podUID="c7497b27-415b-4819-93df-ef43132f4f85" containerName="payload-processing" containerID="cri-o://3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a" gracePeriod=25 Nov 03 17:01:55.567132 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.567108 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9f2868-fsp2c"] Nov 03 17:01:55.616448 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.616403 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" podUID="580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" containerName="payload-pre-processing" containerID="cri-o://821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d" gracePeriod=25 Nov 03 17:01:55.859272 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.859250 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 17:01:55.891826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.891805 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 17:01:55.916687 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.916659 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:55.978128 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.978100 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7497b27-415b-4819-93df-ef43132f4f85-kube-api-access-qqp66\" (UniqueName: \"kubernetes.io/projected/c7497b27-415b-4819-93df-ef43132f4f85-kube-api-access-qqp66\") pod \"c7497b27-415b-4819-93df-ef43132f4f85\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " Nov 03 17:01:55.978328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.978233 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-plugins-config-volume\") pod \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " Nov 03 17:01:55.978328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.978298 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7497b27-415b-4819-93df-ef43132f4f85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7497b27-415b-4819-93df-ef43132f4f85-plugins-config-volume\") pod \"c7497b27-415b-4819-93df-ef43132f4f85\" (UID: \"c7497b27-415b-4819-93df-ef43132f4f85\") " Nov 03 17:01:55.978328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.978319 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-kube-api-access-fgpxb\" (UniqueName: \"kubernetes.io/projected/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-kube-api-access-fgpxb\") pod \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\" (UID: \"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3\") " Nov 03 17:01:55.978681 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.978650 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7497b27-415b-4819-93df-ef43132f4f85-plugins-config-volume" pod "c7497b27-415b-4819-93df-ef43132f4f85" (UID: "c7497b27-415b-4819-93df-ef43132f4f85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:55.978780 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.978660 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-plugins-config-volume" pod "580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" (UID: "580a4eeb-efa9-490f-bb83-8ca3dc7e95a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:01:55.980383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.980365 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7497b27-415b-4819-93df-ef43132f4f85-kube-api-access-qqp66" pod "c7497b27-415b-4819-93df-ef43132f4f85" (UID: "c7497b27-415b-4819-93df-ef43132f4f85"). InnerVolumeSpecName "kube-api-access-qqp66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:55.980463 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:55.980436 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-kube-api-access-fgpxb" pod "580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" (UID: "580a4eeb-efa9-490f-bb83-8ca3dc7e95a3"). InnerVolumeSpecName "kube-api-access-fgpxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:56.003836 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.003799 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429d5bb8-e492-451d-b09f-dea7191ed9ea" path="/var/lib/kubelet/pods/429d5bb8-e492-451d-b09f-dea7191ed9ea/volumes" Nov 03 17:01:56.079118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.079041 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:56.079118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.079067 5946 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7497b27-415b-4819-93df-ef43132f4f85-plugins-config-volume\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:56.079118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.079078 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgpxb\" (UniqueName: \"kubernetes.io/projected/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3-kube-api-access-fgpxb\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:56.079118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.079087 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqp66\" (UniqueName: \"kubernetes.io/projected/c7497b27-415b-4819-93df-ef43132f4f85-kube-api-access-qqp66\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:01:56.147915 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.147882 5946 generic.go:356] "Generic (PLEG): container finished" podID="580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" containerID="821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d" exitCode=0 Nov 03 17:01:56.148116 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.147931 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" Nov 03 17:01:56.148116 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.147977 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" event={"ID":"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3","Type":"ContainerDied","Data":"821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d"} Nov 03 17:01:56.148116 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.148029 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds" event={"ID":"580a4eeb-efa9-490f-bb83-8ca3dc7e95a3","Type":"ContainerDied","Data":"79168c25e91ab81bb7a8f2c5a7ffbf576f981d4f20470d290591e723edba5d74"} Nov 03 17:01:56.148116 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.148055 5946 scope.go:122] "RemoveContainer" containerID="821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d" Nov 03 17:01:56.149490 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.149470 5946 generic.go:356] "Generic (PLEG): container finished" podID="c7497b27-415b-4819-93df-ef43132f4f85" containerID="3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a" exitCode=0 Nov 03 17:01:56.149598 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.149551 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" Nov 03 17:01:56.149598 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.149569 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" event={"ID":"c7497b27-415b-4819-93df-ef43132f4f85","Type":"ContainerDied","Data":"3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a"} Nov 03 17:01:56.149697 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.149605 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks" event={"ID":"c7497b27-415b-4819-93df-ef43132f4f85","Type":"ContainerDied","Data":"187d05eff792186aab1a4126286e6f5655704e86ab7f4434dbd45c5b5a9397db"} Nov 03 17:01:56.157918 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.157901 5946 scope.go:122] "RemoveContainer" containerID="821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d" Nov 03 17:01:56.158188 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:56.158170 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d\": container with ID starting with 821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d not found: ID does not exist" containerID="821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d" Nov 03 17:01:56.158258 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.158197 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d"} err="failed to get container status \"821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d\": rpc error: code = NotFound desc = could not find container \"821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d\": container with ID starting with 821dd172e3ef8f47e1adfcbc4198ab221c24dd60be88c6ceff0001d028869f2d not found: ID does not exist" Nov 03 17:01:56.158258 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.158237 5946 scope.go:122] "RemoveContainer" containerID="3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a" Nov 03 17:01:56.166698 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.166680 5946 scope.go:122] "RemoveContainer" containerID="3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a" Nov 03 17:01:56.166947 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:01:56.166927 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a\": container with ID starting with 3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a not found: ID does not exist" containerID="3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a" Nov 03 17:01:56.166994 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.166953 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a"} err="failed to get container status \"3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a\": rpc error: code = NotFound desc = could not find container \"3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a\": container with ID starting with 3e1a5f9baa5d784eda50bd517efec4af30d7e73ea34be7ba4b31913df45fa75a not found: ID does not exist" Nov 03 17:01:56.324441 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.324407 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds"] Nov 03 17:01:56.331874 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.331809 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9f2868-7b8d9b8955-tkvds"] Nov 03 17:01:56.448218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.448170 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks"] Nov 03 17:01:56.453604 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.453573 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9f2868-8c48c8b59-j5bks"] Nov 03 17:01:56.819462 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:56.819426 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:57.172651 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:57.172542 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:57.366666 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:57.366631 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:58.002838 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:58.002805 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" path="/var/lib/kubelet/pods/580a4eeb-efa9-490f-bb83-8ca3dc7e95a3/volumes" Nov 03 17:01:58.003197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:01:58.003183 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7497b27-415b-4819-93df-ef43132f4f85" path="/var/lib/kubelet/pods/c7497b27-415b-4819-93df-ef43132f4f85/volumes" Nov 03 17:02:01.809596 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:01.809555 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:03.818397 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:03.818361 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:04.006040 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.005995 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" containerID="cri-o://eb5d96c44051df5158a2661310925a61c31d91b748594429ca6d034aaf3a8b64" gracePeriod=45 Nov 03 17:02:04.014024 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.013999 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:04.055587 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.055513 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" containerID="cri-o://6bfa5d7078db55ab34454f18674ae5135e3d6cfc44fa3d2c0066089a565fbfbf" gracePeriod=45 Nov 03 17:02:04.056390 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.056358 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" containerID="cri-o://9e9b9d6226de65ef9f893fbc2efc07f185736ffcdc015d36de8ba1d05d4626b4" gracePeriod=45 Nov 03 17:02:04.118538 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.118505 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl"] Nov 03 17:02:04.118781 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.118742 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" containerID="cri-o://0323f16947daee1b6ec0f5ccd42939be892ed73b41c127bb7e040cd6852db8be" gracePeriod=30 Nov 03 17:02:04.215828 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.215757 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:04.217128 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.217099 5946 generic.go:356] "Generic (PLEG): container finished" podID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerID="9e9b9d6226de65ef9f893fbc2efc07f185736ffcdc015d36de8ba1d05d4626b4" exitCode=0 Nov 03 17:02:04.217299 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.217168 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" event={"ID":"36af74ee-e4d9-4ec5-bbc1-71de63e9d800","Type":"ContainerDied","Data":"9e9b9d6226de65ef9f893fbc2efc07f185736ffcdc015d36de8ba1d05d4626b4"} Nov 03 17:02:04.219248 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.219051 5946 generic.go:356] "Generic (PLEG): container finished" podID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerID="eb5d96c44051df5158a2661310925a61c31d91b748594429ca6d034aaf3a8b64" exitCode=0 Nov 03 17:02:04.219248 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.219158 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" event={"ID":"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a","Type":"ContainerDied","Data":"eb5d96c44051df5158a2661310925a61c31d91b748594429ca6d034aaf3a8b64"} Nov 03 17:02:04.221344 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.221150 5946 generic.go:356] "Generic (PLEG): container finished" podID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerID="6bfa5d7078db55ab34454f18674ae5135e3d6cfc44fa3d2c0066089a565fbfbf" exitCode=0 Nov 03 17:02:04.221344 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.221298 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" event={"ID":"c31c0b4b-ff16-4214-bb48-c3058890db7b","Type":"ContainerDied","Data":"6bfa5d7078db55ab34454f18674ae5135e3d6cfc44fa3d2c0066089a565fbfbf"} Nov 03 17:02:04.333472 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.333446 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 17:02:04.346250 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.346166 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 17:02:04.349434 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.349417 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 17:02:04.369710 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369690 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-home\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-home\") pod \"c31c0b4b-ff16-4214-bb48-c3058890db7b\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " Nov 03 17:02:04.369821 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369746 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-tmp-dir\") pod \"c31c0b4b-ff16-4214-bb48-c3058890db7b\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " Nov 03 17:02:04.369821 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369781 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31c0b4b-ff16-4214-bb48-c3058890db7b-kube-api-access-4q829\" (UniqueName: \"kubernetes.io/projected/c31c0b4b-ff16-4214-bb48-c3058890db7b-kube-api-access-4q829\") pod \"c31c0b4b-ff16-4214-bb48-c3058890db7b\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " Nov 03 17:02:04.369821 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369801 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-model-cache\") pod \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " Nov 03 17:02:04.369821 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369818 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-home\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-home\") pod \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369837 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-model-cache\") pod \"c31c0b4b-ff16-4214-bb48-c3058890db7b\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369868 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tls-certs\") pod \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369894 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-kube-api-access-rl5nt\" (UniqueName: \"kubernetes.io/projected/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-kube-api-access-rl5nt\") pod \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369940 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-dshm\") pod \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369977 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tls-certs\" (UniqueName: \"kubernetes.io/secret/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tls-certs\") pod \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.369965 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-home" pod "c31c0b4b-ff16-4214-bb48-c3058890db7b" (UID: "c31c0b4b-ff16-4214-bb48-c3058890db7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370005 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-dshm\") pod \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " Nov 03 17:02:04.370055 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370033 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tmp-dir\") pod \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370067 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c31c0b4b-ff16-4214-bb48-c3058890db7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c31c0b4b-ff16-4214-bb48-c3058890db7b-tls-certs\") pod \"c31c0b4b-ff16-4214-bb48-c3058890db7b\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370109 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-model-cache\") pod \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\" (UID: \"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a\") " Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370119 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-model-cache" pod "36af74ee-e4d9-4ec5-bbc1-71de63e9d800" (UID: "36af74ee-e4d9-4ec5-bbc1-71de63e9d800"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370127 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-tmp-dir" pod "c31c0b4b-ff16-4214-bb48-c3058890db7b" (UID: "c31c0b4b-ff16-4214-bb48-c3058890db7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370161 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-dshm\") pod \"c31c0b4b-ff16-4214-bb48-c3058890db7b\" (UID: \"c31c0b4b-ff16-4214-bb48-c3058890db7b\") " Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370188 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-home\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-home\") pod \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370251 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-kube-api-access-frrtv\" (UniqueName: \"kubernetes.io/projected/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-kube-api-access-frrtv\") pod \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370292 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tmp-dir\") pod \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\" (UID: \"36af74ee-e4d9-4ec5-bbc1-71de63e9d800\") " Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370392 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-home" pod "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" (UID: "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.370546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370407 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tmp-dir" pod "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" (UID: "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370570 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tmp-dir" pod "36af74ee-e4d9-4ec5-bbc1-71de63e9d800" (UID: "36af74ee-e4d9-4ec5-bbc1-71de63e9d800"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370746 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-home" pod "36af74ee-e4d9-4ec5-bbc1-71de63e9d800" (UID: "36af74ee-e4d9-4ec5-bbc1-71de63e9d800"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370747 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370775 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370791 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370807 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370819 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370830 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.370933 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-model-cache" pod "c31c0b4b-ff16-4214-bb48-c3058890db7b" (UID: "c31c0b4b-ff16-4214-bb48-c3058890db7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.371218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.371252 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-model-cache" pod "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" (UID: "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.373295 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373255 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tls-certs" pod "36af74ee-e4d9-4ec5-bbc1-71de63e9d800" (UID: "36af74ee-e4d9-4ec5-bbc1-71de63e9d800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:04.373295 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373274 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-dshm" pod "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" (UID: "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.373456 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373439 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-kube-api-access-rl5nt" pod "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" (UID: "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a"). InnerVolumeSpecName "kube-api-access-rl5nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:04.373534 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373517 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tls-certs" pod "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" (UID: "cd1b7675-38d0-4487-b70b-8d4b67b5ba5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:04.373589 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373533 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31c0b4b-ff16-4214-bb48-c3058890db7b-kube-api-access-4q829" pod "c31c0b4b-ff16-4214-bb48-c3058890db7b" (UID: "c31c0b4b-ff16-4214-bb48-c3058890db7b"). InnerVolumeSpecName "kube-api-access-4q829". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:04.373721 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373700 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-kube-api-access-frrtv" pod "36af74ee-e4d9-4ec5-bbc1-71de63e9d800" (UID: "36af74ee-e4d9-4ec5-bbc1-71de63e9d800"). InnerVolumeSpecName "kube-api-access-frrtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:04.373819 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373805 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-dshm" pod "36af74ee-e4d9-4ec5-bbc1-71de63e9d800" (UID: "36af74ee-e4d9-4ec5-bbc1-71de63e9d800"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.373944 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.373922 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31c0b4b-ff16-4214-bb48-c3058890db7b-tls-certs" pod "c31c0b4b-ff16-4214-bb48-c3058890db7b" (UID: "c31c0b4b-ff16-4214-bb48-c3058890db7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:04.374696 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.374680 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-dshm" pod "c31c0b4b-ff16-4214-bb48-c3058890db7b" (UID: "c31c0b4b-ff16-4214-bb48-c3058890db7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:04.471665 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471632 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471665 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471663 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471665 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471672 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31c0b4b-ff16-4214-bb48-c3058890db7b-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471680 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471690 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471702 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471714 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrtv\" (UniqueName: \"kubernetes.io/projected/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-kube-api-access-frrtv\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471725 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q829\" (UniqueName: \"kubernetes.io/projected/c31c0b4b-ff16-4214-bb48-c3058890db7b-kube-api-access-4q829\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471734 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31c0b4b-ff16-4214-bb48-c3058890db7b-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471742 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471750 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5nt\" (UniqueName: \"kubernetes.io/projected/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a-kube-api-access-rl5nt\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.471938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.471758 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36af74ee-e4d9-4ec5-bbc1-71de63e9d800-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:04.999692 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:04.999659 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm"] Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000026 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000037 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000046 5946 cpu_manager.go:414] "containerMap: removing container" podUID="f910b760-1ce1-4d3b-81a6-6842ce741f3a" containerName="revoke-keys" Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000052 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f910b760-1ce1-4d3b-81a6-6842ce741f3a" containerName="revoke-keys" Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000071 5946 cpu_manager.go:414] "containerMap: removing container" podUID="c7497b27-415b-4819-93df-ef43132f4f85" containerName="payload-processing" Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000077 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7497b27-415b-4819-93df-ef43132f4f85" containerName="payload-processing" Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000093 5946 cpu_manager.go:414] "containerMap: removing container" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" Nov 03 17:02:05.000102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000098 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000107 5946 cpu_manager.go:414] "containerMap: removing container" podUID="580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" containerName="payload-pre-processing" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000113 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" containerName="payload-pre-processing" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000124 5946 cpu_manager.go:414] "containerMap: removing container" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000129 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000187 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" containerName="main" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000197 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="580a4eeb-efa9-490f-bb83-8ca3dc7e95a3" containerName="payload-pre-processing" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000218 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" containerName="main" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000226 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" containerName="main" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000231 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="f910b760-1ce1-4d3b-81a6-6842ce741f3a" containerName="revoke-keys" Nov 03 17:02:05.000383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.000239 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7497b27-415b-4819-93df-ef43132f4f85" containerName="payload-processing" Nov 03 17:02:05.003028 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.003012 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.060072 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.060034 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:02:05.062117 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.062062 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:02:05.062354 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.062329 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv"] Nov 03 17:02:05.064820 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.064790 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm"] Nov 03 17:02:05.077334 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.077306 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.077486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.077465 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.077566 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.077492 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6g2p\" (UniqueName: \"kubernetes.io/projected/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-kube-api-access-k6g2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.110953 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.110921 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:02:05.110953 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.110949 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:02:05.113145 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.113122 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0192f3b7-qc4fv"] Nov 03 17:02:05.178728 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.178690 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.178728 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.178729 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6g2p\" (UniqueName: \"kubernetes.io/projected/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-kube-api-access-k6g2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.178953 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.178753 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.179417 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.179393 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.181195 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.181175 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.228156 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.228122 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" Nov 03 17:02:05.228373 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.228129 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54" event={"ID":"36af74ee-e4d9-4ec5-bbc1-71de63e9d800","Type":"ContainerDied","Data":"4a6a7cffced73e5181950d9ff591395e0659e25c19c3df1aeafa94b5a4c72be7"} Nov 03 17:02:05.228373 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.228278 5946 scope.go:122] "RemoveContainer" containerID="9e9b9d6226de65ef9f893fbc2efc07f185736ffcdc015d36de8ba1d05d4626b4" Nov 03 17:02:05.229810 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.229786 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" event={"ID":"cd1b7675-38d0-4487-b70b-8d4b67b5ba5a","Type":"ContainerDied","Data":"c623f7237931a6ab4867ad9f5e08a59a65133919a7326a3834ff87104357cdbc"} Nov 03 17:02:05.229810 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.229801 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6" Nov 03 17:02:05.231401 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.231377 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" event={"ID":"c31c0b4b-ff16-4214-bb48-c3058890db7b","Type":"ContainerDied","Data":"28d2232e77d9bbbe5a1a462f3b1aa031de06eb2377422244546a52c2ce9a183b"} Nov 03 17:02:05.231529 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.231416 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg" Nov 03 17:02:05.239766 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.239741 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6g2p\" (UniqueName: \"kubernetes.io/projected/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-kube-api-access-k6g2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.240001 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.239986 5946 scope.go:122] "RemoveContainer" containerID="eb5d96c44051df5158a2661310925a61c31d91b748594429ca6d034aaf3a8b64" Nov 03 17:02:05.254358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.254340 5946 scope.go:122] "RemoveContainer" containerID="6bfa5d7078db55ab34454f18674ae5135e3d6cfc44fa3d2c0066089a565fbfbf" Nov 03 17:02:05.315314 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.315282 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:05.401301 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.401266 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg"] Nov 03 17:02:05.407480 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.407453 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/premium-w0-42da0f-kserve-844bcbddff-rx2cg"] Nov 03 17:02:05.523601 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.523517 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54"] Nov 03 17:02:05.528598 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.528570 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/simulator-w0-42da0f-kserve-6cc86cd8f8-bng54"] Nov 03 17:02:05.644880 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.644842 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6"] Nov 03 17:02:05.653886 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.653857 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-w0-42da0f-kserve-569d787f6b-85jr6"] Nov 03 17:02:05.698257 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:05.698231 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm"] Nov 03 17:02:05.699872 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:02:05.699833 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1702d0_3de3_45a7_a4e2_00f05f2cf800.slice/crio-f9bbe7e231e2499573998c62e774d4f773460f31e101df6633a79c9d1f77c063 WatchSource:0}: Error finding container f9bbe7e231e2499573998c62e774d4f773460f31e101df6633a79c9d1f77c063: Status 404 returned error can't find the container with id f9bbe7e231e2499573998c62e774d4f773460f31e101df6633a79c9d1f77c063 Nov 03 17:02:06.005057 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.005022 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36af74ee-e4d9-4ec5-bbc1-71de63e9d800" path="/var/lib/kubelet/pods/36af74ee-e4d9-4ec5-bbc1-71de63e9d800/volumes" Nov 03 17:02:06.005682 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.005661 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31c0b4b-ff16-4214-bb48-c3058890db7b" path="/var/lib/kubelet/pods/c31c0b4b-ff16-4214-bb48-c3058890db7b/volumes" Nov 03 17:02:06.006181 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.006164 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb487847-275d-4129-9a99-29439ac16a6f" path="/var/lib/kubelet/pods/cb487847-275d-4129-9a99-29439ac16a6f/volumes" Nov 03 17:02:06.007004 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.006988 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1b7675-38d0-4487-b70b-8d4b67b5ba5a" path="/var/lib/kubelet/pods/cd1b7675-38d0-4487-b70b-8d4b67b5ba5a/volumes" Nov 03 17:02:06.110174 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.110136 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr"] Nov 03 17:02:06.113136 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.113119 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.175275 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.175239 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr"] Nov 03 17:02:06.187918 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.187843 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.187918 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.187878 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrdxl\" (UniqueName: \"kubernetes.io/projected/09a6c1d2-abc9-491b-9317-edba35b8ab4d-kube-api-access-zrdxl\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.188098 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.188032 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.243721 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.243687 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" event={"ID":"7f1702d0-3de3-45a7-a4e2-00f05f2cf800","Type":"ContainerStarted","Data":"0abb846d49841e092f1f4245390a78c6c6fed1a83770b0b33e3c37f00ab25444"} Nov 03 17:02:06.243901 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.243741 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" event={"ID":"7f1702d0-3de3-45a7-a4e2-00f05f2cf800","Type":"ContainerStarted","Data":"f9bbe7e231e2499573998c62e774d4f773460f31e101df6633a79c9d1f77c063"} Nov 03 17:02:06.289350 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.289317 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.289517 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.289373 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.289517 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.289404 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrdxl\" (UniqueName: \"kubernetes.io/projected/09a6c1d2-abc9-491b-9317-edba35b8ab4d-kube-api-access-zrdxl\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.290077 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.290057 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.291966 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.291945 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.349840 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.349809 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrdxl\" (UniqueName: \"kubernetes.io/projected/09a6c1d2-abc9-491b-9317-edba35b8ab4d-kube-api-access-zrdxl\") pod \"maas-api-revoke-sub-82d48023-8rlmr\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.364933 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.364884 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" podStartSLOduration=2.3648701 podStartE2EDuration="2.3648701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:06.364135351 +0000 UTC m=+2996.485188175" watchObservedRunningTime="2026-11-03 17:02:06.3648701 +0000 UTC m=+2996.485922923" Nov 03 17:02:06.425675 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.425641 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:06.803371 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.803345 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr"] Nov 03 17:02:06.805121 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:02:06.805094 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a6c1d2_abc9_491b_9317_edba35b8ab4d.slice/crio-ad508e55e50a2a7d71b62b2fb839197ade5b8b9169a87f969a44b9a29d1777d7 WatchSource:0}: Error finding container ad508e55e50a2a7d71b62b2fb839197ade5b8b9169a87f969a44b9a29d1777d7: Status 404 returned error can't find the container with id ad508e55e50a2a7d71b62b2fb839197ade5b8b9169a87f969a44b9a29d1777d7 Nov 03 17:02:06.990286 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:06.990249 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:07.251858 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:07.251823 5946 generic.go:356] "Generic (PLEG): container finished" podID="7f1702d0-3de3-45a7-a4e2-00f05f2cf800" containerID="0abb846d49841e092f1f4245390a78c6c6fed1a83770b0b33e3c37f00ab25444" exitCode=0 Nov 03 17:02:07.252326 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:07.251910 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" event={"ID":"7f1702d0-3de3-45a7-a4e2-00f05f2cf800","Type":"ContainerDied","Data":"0abb846d49841e092f1f4245390a78c6c6fed1a83770b0b33e3c37f00ab25444"} Nov 03 17:02:07.253542 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:07.253519 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" event={"ID":"09a6c1d2-abc9-491b-9317-edba35b8ab4d","Type":"ContainerStarted","Data":"10e6440f2b81ec4a042a2265389395c77a297b0fdedd8451bd87d0ab7691398f"} Nov 03 17:02:07.253667 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:07.253564 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" event={"ID":"09a6c1d2-abc9-491b-9317-edba35b8ab4d","Type":"ContainerStarted","Data":"ad508e55e50a2a7d71b62b2fb839197ade5b8b9169a87f969a44b9a29d1777d7"} Nov 03 17:02:07.500052 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:07.499994 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" podStartSLOduration=1.49997814 podStartE2EDuration="1.49997814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:07.49785476 +0000 UTC m=+2997.618907609" watchObservedRunningTime="2026-11-03 17:02:07.49997814 +0000 UTC m=+2997.621030963" Nov 03 17:02:08.261361 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.261317 5946 generic.go:356] "Generic (PLEG): container finished" podID="09a6c1d2-abc9-491b-9317-edba35b8ab4d" containerID="10e6440f2b81ec4a042a2265389395c77a297b0fdedd8451bd87d0ab7691398f" exitCode=0 Nov 03 17:02:08.261836 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.261406 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" event={"ID":"09a6c1d2-abc9-491b-9317-edba35b8ab4d","Type":"ContainerDied","Data":"10e6440f2b81ec4a042a2265389395c77a297b0fdedd8451bd87d0ab7691398f"} Nov 03 17:02:08.402293 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.402265 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:08.509889 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.509851 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-xks-ca\") pod \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " Nov 03 17:02:08.510066 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.509928 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-openshift-ca\") pod \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " Nov 03 17:02:08.510066 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.509999 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-kube-api-access-k6g2p\" (UniqueName: \"kubernetes.io/projected/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-kube-api-access-k6g2p\") pod \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\" (UID: \"7f1702d0-3de3-45a7-a4e2-00f05f2cf800\") " Nov 03 17:02:08.510362 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.510336 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-openshift-ca" pod "7f1702d0-3de3-45a7-a4e2-00f05f2cf800" (UID: "7f1702d0-3de3-45a7-a4e2-00f05f2cf800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:02:08.512133 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.512070 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-xks-ca" pod "7f1702d0-3de3-45a7-a4e2-00f05f2cf800" (UID: "7f1702d0-3de3-45a7-a4e2-00f05f2cf800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:08.512329 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.512309 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-kube-api-access-k6g2p" pod "7f1702d0-3de3-45a7-a4e2-00f05f2cf800" (UID: "7f1702d0-3de3-45a7-a4e2-00f05f2cf800"). InnerVolumeSpecName "kube-api-access-k6g2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:08.611020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.610990 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6g2p\" (UniqueName: \"kubernetes.io/projected/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-kube-api-access-k6g2p\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:08.611020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.611019 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:08.611239 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:08.611029 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1702d0-3de3-45a7-a4e2-00f05f2cf800-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.269252 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.269199 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" Nov 03 17:02:09.269252 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.269231 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm" event={"ID":"7f1702d0-3de3-45a7-a4e2-00f05f2cf800","Type":"ContainerDied","Data":"f9bbe7e231e2499573998c62e774d4f773460f31e101df6633a79c9d1f77c063"} Nov 03 17:02:09.269815 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.269289 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9bbe7e231e2499573998c62e774d4f773460f31e101df6633a79c9d1f77c063" Nov 03 17:02:09.271535 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.271509 5946 generic.go:356] "Generic (PLEG): container finished" podID="523fd694-634d-4bd0-b573-773d9e42feeb" containerID="0323f16947daee1b6ec0f5ccd42939be892ed73b41c127bb7e040cd6852db8be" exitCode=0 Nov 03 17:02:09.271814 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.271798 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" event={"ID":"523fd694-634d-4bd0-b573-773d9e42feeb","Type":"ContainerDied","Data":"0323f16947daee1b6ec0f5ccd42939be892ed73b41c127bb7e040cd6852db8be"} Nov 03 17:02:09.271890 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.271836 5946 scope.go:122] "RemoveContainer" containerID="7c1123317bc485eb77f316f2bd3c225dc026166e13a7605f2b48f96b66f0d053" Nov 03 17:02:09.363531 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.363506 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 17:02:09.417404 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417378 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-credential-socket\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417533 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417430 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/523fd694-634d-4bd0-b573-773d9e42feeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523fd694-634d-4bd0-b573-773d9e42feeb-istio-podinfo\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417533 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417469 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-istio-token\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-istio-token\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417533 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417519 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-kube-api-access-6c6p4\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-kube-api-access-6c6p4\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417734 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417555 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-envoy\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417734 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417643 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-credential-socket" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:09.417734 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417664 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523fd694-634d-4bd0-b573-773d9e42feeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523fd694-634d-4bd0-b573-773d9e42feeb-istiod-ca-cert\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417734 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417691 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-data\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417941 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417738 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-certs\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.417941 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.417767 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-socket\") pod \"523fd694-634d-4bd0-b573-773d9e42feeb\" (UID: \"523fd694-634d-4bd0-b573-773d9e42feeb\") " Nov 03 17:02:09.418486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.418076 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.418486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.418124 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523fd694-634d-4bd0-b573-773d9e42feeb-istiod-ca-cert" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:02:09.418640 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.418483 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-data" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:09.418640 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.418495 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-socket" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:09.418796 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.418763 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-certs" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:09.419944 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.419918 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/523fd694-634d-4bd0-b573-773d9e42feeb-istio-podinfo" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:02:09.420060 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.419994 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-istio-token" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:09.420228 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.420175 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-kube-api-access-6c6p4" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "kube-api-access-6c6p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:09.420355 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.420312 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:09.420999 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.420978 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-envoy" pod "523fd694-634d-4bd0-b573-773d9e42feeb" (UID: "523fd694-634d-4bd0-b573-773d9e42feeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:09.518696 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.518657 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a6c1d2-abc9-491b-9317-edba35b8ab4d-kube-api-access-zrdxl\" (UniqueName: \"kubernetes.io/projected/09a6c1d2-abc9-491b-9317-edba35b8ab4d-kube-api-access-zrdxl\") pod \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " Nov 03 17:02:09.518900 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.518700 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-openshift-ca\") pod \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " Nov 03 17:02:09.518900 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.518834 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-xks-ca\") pod \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\" (UID: \"09a6c1d2-abc9-491b-9317-edba35b8ab4d\") " Nov 03 17:02:09.519060 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519039 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-openshift-ca" pod "09a6c1d2-abc9-491b-9317-edba35b8ab4d" (UID: "09a6c1d2-abc9-491b-9317-edba35b8ab4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:02:09.519297 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519276 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523fd694-634d-4bd0-b573-773d9e42feeb-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519299 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519309 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6p4\" (UniqueName: \"kubernetes.io/projected/523fd694-634d-4bd0-b573-773d9e42feeb-kube-api-access-6c6p4\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519319 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519328 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519336 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523fd694-634d-4bd0-b573-773d9e42feeb-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519344 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519352 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.519392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.519361 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523fd694-634d-4bd0-b573-773d9e42feeb-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.520961 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.520937 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-xks-ca" pod "09a6c1d2-abc9-491b-9317-edba35b8ab4d" (UID: "09a6c1d2-abc9-491b-9317-edba35b8ab4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:09.521061 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.521003 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a6c1d2-abc9-491b-9317-edba35b8ab4d-kube-api-access-zrdxl" pod "09a6c1d2-abc9-491b-9317-edba35b8ab4d" (UID: "09a6c1d2-abc9-491b-9317-edba35b8ab4d"). InnerVolumeSpecName "kube-api-access-zrdxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:09.620458 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.620428 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrdxl\" (UniqueName: \"kubernetes.io/projected/09a6c1d2-abc9-491b-9317-edba35b8ab4d-kube-api-access-zrdxl\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:09.620458 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:09.620455 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a6c1d2-abc9-491b-9317-edba35b8ab4d-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:10.278969 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.278880 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" Nov 03 17:02:10.278969 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.278918 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl" event={"ID":"523fd694-634d-4bd0-b573-773d9e42feeb","Type":"ContainerDied","Data":"5f38918fbe858781c462f94158a12e6bd859d4efadb209261943a0a758de44a4"} Nov 03 17:02:10.279548 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.278984 5946 scope.go:122] "RemoveContainer" containerID="0323f16947daee1b6ec0f5ccd42939be892ed73b41c127bb7e040cd6852db8be" Nov 03 17:02:10.280409 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.280389 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" Nov 03 17:02:10.280533 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.280419 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr" event={"ID":"09a6c1d2-abc9-491b-9317-edba35b8ab4d","Type":"ContainerDied","Data":"ad508e55e50a2a7d71b62b2fb839197ade5b8b9169a87f969a44b9a29d1777d7"} Nov 03 17:02:10.280533 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.280453 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad508e55e50a2a7d71b62b2fb839197ade5b8b9169a87f969a44b9a29d1777d7" Nov 03 17:02:10.509676 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.509652 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:02:10.520270 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.520248 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:02:10.562830 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.562738 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl"] Nov 03 17:02:10.612487 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:10.612453 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9f2868-openshift-default-54cdb6696f-k74wl"] Nov 03 17:02:12.003099 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:12.003069 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" path="/var/lib/kubelet/pods/523fd694-634d-4bd0-b573-773d9e42feeb/volumes" Nov 03 17:02:12.120714 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:12.120681 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:12.584801 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:02:12.584764 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:02:13.509376 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:13.509333 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:02:16.852520 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:16.852482 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:17.057449 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:17.057416 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:20.064330 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:20.064294 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7"] Nov 03 17:02:20.068348 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:20.068326 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b830e813-2mrw7"] Nov 03 17:02:21.063308 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:21.063273 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:22.004869 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:22.004840 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51221a7-c981-4038-aac4-2211ecc22195" path="/var/lib/kubelet/pods/a51221a7-c981-4038-aac4-2211ecc22195/volumes" Nov 03 17:02:22.161912 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:22.161863 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:22.508460 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:22.508421 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:22.777290 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:22.777162 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:22.998902 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:22.998865 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9"] Nov 03 17:02:23.049360 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:23.049285 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42da0f-2w2b9"] Nov 03 17:02:23.516814 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:23.516771 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:23.731352 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:23.731312 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:23.998108 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:23.998078 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:24.002911 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:24.002884 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f910b760-1ce1-4d3b-81a6-6842ce741f3a" path="/var/lib/kubelet/pods/f910b760-1ce1-4d3b-81a6-6842ce741f3a/volumes" Nov 03 17:02:24.208460 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:24.208419 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm"] Nov 03 17:02:24.228893 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:24.228838 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:24.259387 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:24.259322 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-509f14-9g8rm"] Nov 03 17:02:24.633320 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:24.633191 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:26.002548 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:26.002518 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1702d0-3de3-45a7-a4e2-00f05f2cf800" path="/var/lib/kubelet/pods/7f1702d0-3de3-45a7-a4e2-00f05f2cf800/volumes" Nov 03 17:02:28.242519 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:28.242484 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:28.461422 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:28.461386 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:29.163498 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:29.163458 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:29.768772 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:29.768701 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:32.562848 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:32.562809 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:32.841975 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:32.841859 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:33.265444 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:33.265387 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:34.062582 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:34.062530 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:34.308114 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:34.308077 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:34.372776 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:34.372688 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq"] Nov 03 17:02:34.372953 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:34.372897 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" containerID="cri-o://be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0" gracePeriod=30 Nov 03 17:02:34.658240 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:34.658137 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:34.890148 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:34.890112 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:35.357448 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357408 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs"] Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357798 5946 cpu_manager.go:414] "containerMap: removing container" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357810 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357821 5946 cpu_manager.go:414] "containerMap: removing container" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357826 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357837 5946 cpu_manager.go:414] "containerMap: removing container" podUID="09a6c1d2-abc9-491b-9317-edba35b8ab4d" containerName="revoke-keys" Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357843 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a6c1d2-abc9-491b-9317-edba35b8ab4d" containerName="revoke-keys" Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357871 5946 cpu_manager.go:414] "containerMap: removing container" podUID="7f1702d0-3de3-45a7-a4e2-00f05f2cf800" containerName="revoke-keys" Nov 03 17:02:35.357897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357875 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1702d0-3de3-45a7-a4e2-00f05f2cf800" containerName="revoke-keys" Nov 03 17:02:35.358234 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357933 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1702d0-3de3-45a7-a4e2-00f05f2cf800" containerName="revoke-keys" Nov 03 17:02:35.358234 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357945 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" Nov 03 17:02:35.358234 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357951 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="523fd694-634d-4bd0-b573-773d9e42feeb" containerName="istio-proxy" Nov 03 17:02:35.358234 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.357956 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a6c1d2-abc9-491b-9317-edba35b8ab4d" containerName="revoke-keys" Nov 03 17:02:35.360001 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.359985 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.414423 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.414390 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:02:35.465515 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.465476 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:02:35.465850 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.465829 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:02:35.465977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.465950 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:02:35.467739 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.467718 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs"] Nov 03 17:02:35.468062 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.468046 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.468106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.468075 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtt8\" (UniqueName: \"kubernetes.io/projected/d92cfa34-6405-44c9-b375-734cf71af873-kube-api-access-gmtt8\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.468106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.468101 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.569393 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.569360 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.569393 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.569396 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtt8\" (UniqueName: \"kubernetes.io/projected/d92cfa34-6405-44c9-b375-734cf71af873-kube-api-access-gmtt8\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.569621 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.569423 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.570020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.569992 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.571892 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.571868 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.630989 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.630928 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtt8\" (UniqueName: \"kubernetes.io/projected/d92cfa34-6405-44c9-b375-734cf71af873-kube-api-access-gmtt8\") pod \"maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:35.672034 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:35.672004 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:36.054254 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:36.054228 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs"] Nov 03 17:02:36.056544 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:02:36.056517 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92cfa34_6405_44c9_b375_734cf71af873.slice/crio-ff81e1590da90292d41ad829a9307543025718decd75f5182eeee36594fc603b WatchSource:0}: Error finding container ff81e1590da90292d41ad829a9307543025718decd75f5182eeee36594fc603b: Status 404 returned error can't find the container with id ff81e1590da90292d41ad829a9307543025718decd75f5182eeee36594fc603b Nov 03 17:02:36.440008 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:36.439958 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" event={"ID":"d92cfa34-6405-44c9-b375-734cf71af873","Type":"ContainerStarted","Data":"db0ed2e16bc08374ea9d634663a1bd95d258ceffd8cb8f61cfb01ea4e55e05b5"} Nov 03 17:02:36.440508 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:36.440023 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" event={"ID":"d92cfa34-6405-44c9-b375-734cf71af873","Type":"ContainerStarted","Data":"ff81e1590da90292d41ad829a9307543025718decd75f5182eeee36594fc603b"} Nov 03 17:02:36.561532 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:36.561469 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" podStartSLOduration=1.5614540460000002 podStartE2EDuration="1.561454046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:36.56069901 +0000 UTC m=+3026.681751846" watchObservedRunningTime="2026-11-03 17:02:36.561454046 +0000 UTC m=+3026.682506916" Nov 03 17:02:37.447880 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:37.447842 5946 generic.go:356] "Generic (PLEG): container finished" podID="d92cfa34-6405-44c9-b375-734cf71af873" containerID="db0ed2e16bc08374ea9d634663a1bd95d258ceffd8cb8f61cfb01ea4e55e05b5" exitCode=0 Nov 03 17:02:37.448349 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:37.447935 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" event={"ID":"d92cfa34-6405-44c9-b375-734cf71af873","Type":"ContainerDied","Data":"db0ed2e16bc08374ea9d634663a1bd95d258ceffd8cb8f61cfb01ea4e55e05b5"} Nov 03 17:02:38.360499 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.360447 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:38.601344 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.601310 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:38.700256 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.700149 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-xks-ca\") pod \"d92cfa34-6405-44c9-b375-734cf71af873\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " Nov 03 17:02:38.700398 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.700260 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-openshift-ca\") pod \"d92cfa34-6405-44c9-b375-734cf71af873\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " Nov 03 17:02:38.700398 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.700320 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92cfa34-6405-44c9-b375-734cf71af873-kube-api-access-gmtt8\" (UniqueName: \"kubernetes.io/projected/d92cfa34-6405-44c9-b375-734cf71af873-kube-api-access-gmtt8\") pod \"d92cfa34-6405-44c9-b375-734cf71af873\" (UID: \"d92cfa34-6405-44c9-b375-734cf71af873\") " Nov 03 17:02:38.700689 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.700658 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-openshift-ca" pod "d92cfa34-6405-44c9-b375-734cf71af873" (UID: "d92cfa34-6405-44c9-b375-734cf71af873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:02:38.702390 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.702364 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-xks-ca" pod "d92cfa34-6405-44c9-b375-734cf71af873" (UID: "d92cfa34-6405-44c9-b375-734cf71af873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:38.702634 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.702612 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92cfa34-6405-44c9-b375-734cf71af873-kube-api-access-gmtt8" pod "d92cfa34-6405-44c9-b375-734cf71af873" (UID: "d92cfa34-6405-44c9-b375-734cf71af873"). InnerVolumeSpecName "kube-api-access-gmtt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:38.801653 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.801623 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:38.801653 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.801650 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92cfa34-6405-44c9-b375-734cf71af873-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:38.801860 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:38.801660 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmtt8\" (UniqueName: \"kubernetes.io/projected/d92cfa34-6405-44c9-b375-734cf71af873-kube-api-access-gmtt8\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.318720 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.318686 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:39.462275 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.462237 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" event={"ID":"d92cfa34-6405-44c9-b375-734cf71af873","Type":"ContainerDied","Data":"ff81e1590da90292d41ad829a9307543025718decd75f5182eeee36594fc603b"} Nov 03 17:02:39.462275 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.462259 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs" Nov 03 17:02:39.462490 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.462292 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff81e1590da90292d41ad829a9307543025718decd75f5182eeee36594fc603b" Nov 03 17:02:39.625024 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.625001 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 17:02:39.776302 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.776268 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x"] Nov 03 17:02:39.812200 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812156 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-token\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-token\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812433 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812225 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-credential-socket\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812433 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812278 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-podinfo\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812433 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812316 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267d739d-4e26-432a-bd7f-865b6ccbc8df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267d739d-4e26-432a-bd7f-865b6ccbc8df-istiod-ca-cert\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812433 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812357 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-socket\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812433 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812386 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-certs\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812433 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812423 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-kube-api-access-v2dt5\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-kube-api-access-v2dt5\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812486 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-data\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812514 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-envoy\") pod \"267d739d-4e26-432a-bd7f-865b6ccbc8df\" (UID: \"267d739d-4e26-432a-bd7f-865b6ccbc8df\") " Nov 03 17:02:39.812846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812591 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-credential-socket" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:39.812846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812682 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267d739d-4e26-432a-bd7f-865b6ccbc8df-istiod-ca-cert" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:02:39.813230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812883 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-socket" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:39.813230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812907 5946 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-credential-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.813230 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.812929 5946 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267d739d-4e26-432a-bd7f-865b6ccbc8df-istiod-ca-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.813995 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.813404 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-certs" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:39.813995 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.813570 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-data" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:39.814978 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.814951 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-podinfo" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:02:39.815073 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.814997 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-kube-api-access-v2dt5" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "kube-api-access-v2dt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:39.815173 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.815153 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-token" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:39.815395 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.815378 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-envoy" pod "267d739d-4e26-432a-bd7f-865b6ccbc8df" (UID: "267d739d-4e26-432a-bd7f-865b6ccbc8df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:39.913469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.913396 5946 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-token\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.913469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.913423 5946 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-podinfo\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.913469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.913432 5946 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-socket\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.913469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.913449 5946 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-workload-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.913469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.913460 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2dt5\" (UniqueName: \"kubernetes.io/projected/267d739d-4e26-432a-bd7f-865b6ccbc8df-kube-api-access-v2dt5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.913469 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.913471 5946 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-data\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.913813 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.913480 5946 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267d739d-4e26-432a-bd7f-865b6ccbc8df-istio-envoy\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:39.938319 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:39.938290 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:40.471282 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.471244 5946 generic.go:356] "Generic (PLEG): container finished" podID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerID="be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0" exitCode=0 Nov 03 17:02:40.471282 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.471278 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" Nov 03 17:02:40.471544 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.471310 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" event={"ID":"267d739d-4e26-432a-bd7f-865b6ccbc8df","Type":"ContainerDied","Data":"be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0"} Nov 03 17:02:40.471544 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.471372 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq" event={"ID":"267d739d-4e26-432a-bd7f-865b6ccbc8df","Type":"ContainerDied","Data":"d322b9f0bb0dc5fc208986a22fc54aa18722400e9252060d3db3836244ea1602"} Nov 03 17:02:40.471544 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.471396 5946 scope.go:122] "RemoveContainer" containerID="be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0" Nov 03 17:02:40.481393 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.481366 5946 scope.go:122] "RemoveContainer" containerID="e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c" Nov 03 17:02:40.490054 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.490031 5946 scope.go:122] "RemoveContainer" containerID="be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0" Nov 03 17:02:40.490325 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:02:40.490305 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0\": container with ID starting with be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0 not found: ID does not exist" containerID="be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0" Nov 03 17:02:40.490392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.490332 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0"} err="failed to get container status \"be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0\": rpc error: code = NotFound desc = could not find container \"be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0\": container with ID starting with be0d92fb7ce67418de0b69541b41a947cb6ad304c139d4228f8c49cdb222bfb0 not found: ID does not exist" Nov 03 17:02:40.490392 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.490350 5946 scope.go:122] "RemoveContainer" containerID="e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c" Nov 03 17:02:40.490595 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:02:40.490574 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c\": container with ID starting with e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c not found: ID does not exist" containerID="e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c" Nov 03 17:02:40.490636 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.490603 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c"} err="failed to get container status \"e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c\": rpc error: code = NotFound desc = could not find container \"e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c\": container with ID starting with e9ff41767081327f7ab37c681c26b0ccbb810fe894858e9bde89db4f9ce1248c not found: ID does not exist" Nov 03 17:02:40.563427 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.563394 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:02:40.644963 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.644919 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq"] Nov 03 17:02:40.651782 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:40.651756 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-509f14-openshift-default-9f649ffdc-hpfhq"] Nov 03 17:02:41.538870 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:41.538834 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:42.003426 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:42.003392 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" path="/var/lib/kubelet/pods/267d739d-4e26-432a-bd7f-865b6ccbc8df/volumes" Nov 03 17:02:42.585007 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:02:42.584967 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:02:44.134525 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:44.134427 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:45.204799 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:45.204745 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:47.295637 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:47.295594 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:47.544155 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:47.544123 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:49.006477 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:49.006444 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:53.194281 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194243 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc"] Nov 03 17:02:53.194691 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194656 5946 cpu_manager.go:414] "containerMap: removing container" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" Nov 03 17:02:53.194691 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194671 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" Nov 03 17:02:53.194691 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194685 5946 cpu_manager.go:414] "containerMap: removing container" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" Nov 03 17:02:53.194691 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194691 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" Nov 03 17:02:53.194826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194707 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d92cfa34-6405-44c9-b375-734cf71af873" containerName="revoke-keys" Nov 03 17:02:53.194826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194713 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92cfa34-6405-44c9-b375-734cf71af873" containerName="revoke-keys" Nov 03 17:02:53.194826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194798 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" Nov 03 17:02:53.194826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194809 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="267d739d-4e26-432a-bd7f-865b6ccbc8df" containerName="istio-proxy" Nov 03 17:02:53.194826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.194817 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92cfa34-6405-44c9-b375-734cf71af873" containerName="revoke-keys" Nov 03 17:02:53.198415 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.198394 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.240477 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.240442 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.240668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.240512 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfckd\" (UniqueName: \"kubernetes.io/projected/3b201fc4-3161-44bd-aec1-eea4b7748f5e-kube-api-access-zfckd\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.240668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.240605 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.255872 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.255836 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:02:53.260275 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.260236 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc"] Nov 03 17:02:53.306923 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.306887 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:02:53.307102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.306949 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:02:53.342062 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.342020 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.342249 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.342117 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.342249 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.342236 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfckd\" (UniqueName: \"kubernetes.io/projected/3b201fc4-3161-44bd-aec1-eea4b7748f5e-kube-api-access-zfckd\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.342723 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.342705 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.355798 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.355764 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:02:53.366033 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.365993 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.402528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.402500 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfckd\" (UniqueName: \"kubernetes.io/projected/3b201fc4-3161-44bd-aec1-eea4b7748f5e-kube-api-access-zfckd\") pod \"maas-api-revoke-sub-1dd062fd-vm9zc\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.510736 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.510700 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:53.899545 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:53.899517 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc"] Nov 03 17:02:53.901336 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:02:53.901309 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b201fc4_3161_44bd_aec1_eea4b7748f5e.slice/crio-dbaf5705f7fcd6793c7e87516ee2123fdf8a3732c9444d148ea712686f492dd9 WatchSource:0}: Error finding container dbaf5705f7fcd6793c7e87516ee2123fdf8a3732c9444d148ea712686f492dd9: Status 404 returned error can't find the container with id dbaf5705f7fcd6793c7e87516ee2123fdf8a3732c9444d148ea712686f492dd9 Nov 03 17:02:54.558797 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:54.558761 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" event={"ID":"3b201fc4-3161-44bd-aec1-eea4b7748f5e","Type":"ContainerStarted","Data":"3d481c6126093f5143899de52285801771d0a0356518343a266c859d8d08f85b"} Nov 03 17:02:54.559276 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:54.558819 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" event={"ID":"3b201fc4-3161-44bd-aec1-eea4b7748f5e","Type":"ContainerStarted","Data":"dbaf5705f7fcd6793c7e87516ee2123fdf8a3732c9444d148ea712686f492dd9"} Nov 03 17:02:54.680163 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:54.680100 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" podStartSLOduration=1.680081946 podStartE2EDuration="1.680081946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:54.679576279 +0000 UTC m=+3044.800629099" watchObservedRunningTime="2026-11-03 17:02:54.680081946 +0000 UTC m=+3044.801134770" Nov 03 17:02:54.797063 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:54.797026 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" containerID="cri-o://c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb" gracePeriod=45 Nov 03 17:02:55.049477 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.049451 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 17:02:55.159371 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159275 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-dshm\") pod \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " Nov 03 17:02:55.159371 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159344 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tls-certs\" (UniqueName: \"kubernetes.io/secret/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tls-certs\") pod \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " Nov 03 17:02:55.159619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159396 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tmp-dir\") pod \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " Nov 03 17:02:55.159619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159453 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-model-cache\") pod \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " Nov 03 17:02:55.159619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159483 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-kube-api-access-t7wrm\" (UniqueName: \"kubernetes.io/projected/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-kube-api-access-t7wrm\") pod \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " Nov 03 17:02:55.159619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159519 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-home\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-home\") pod \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\" (UID: \"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44\") " Nov 03 17:02:55.159826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159701 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tmp-dir" pod "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" (UID: "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:55.159826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159735 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-model-cache" pod "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" (UID: "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:55.159826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159794 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-home" pod "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" (UID: "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:55.159985 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159928 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:55.159985 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159947 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:55.159985 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.159961 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:55.161765 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.161742 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-dshm" pod "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" (UID: "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:02:55.161862 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.161785 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tls-certs" pod "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" (UID: "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:55.161862 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.161824 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-kube-api-access-t7wrm" pod "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" (UID: "12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44"). InnerVolumeSpecName "kube-api-access-t7wrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:55.261394 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.261360 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:55.261394 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.261390 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:55.261600 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.261403 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wrm\" (UniqueName: \"kubernetes.io/projected/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44-kube-api-access-t7wrm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:55.566107 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.566066 5946 generic.go:356] "Generic (PLEG): container finished" podID="3b201fc4-3161-44bd-aec1-eea4b7748f5e" containerID="3d481c6126093f5143899de52285801771d0a0356518343a266c859d8d08f85b" exitCode=0 Nov 03 17:02:55.566614 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.566157 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" event={"ID":"3b201fc4-3161-44bd-aec1-eea4b7748f5e","Type":"ContainerDied","Data":"3d481c6126093f5143899de52285801771d0a0356518343a266c859d8d08f85b"} Nov 03 17:02:55.567519 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.567497 5946 generic.go:356] "Generic (PLEG): container finished" podID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerID="c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb" exitCode=0 Nov 03 17:02:55.567623 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.567541 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" Nov 03 17:02:55.567623 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.567580 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" event={"ID":"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44","Type":"ContainerDied","Data":"c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb"} Nov 03 17:02:55.567623 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.567618 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x" event={"ID":"12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44","Type":"ContainerDied","Data":"0ae35160db49542b6f8d31450de7bfdf34a53d33e90c615a8b387a06db65893d"} Nov 03 17:02:55.567729 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.567636 5946 scope.go:122] "RemoveContainer" containerID="c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb" Nov 03 17:02:55.577642 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.577626 5946 scope.go:122] "RemoveContainer" containerID="c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb" Nov 03 17:02:55.577915 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:02:55.577894 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb\": container with ID starting with c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb not found: ID does not exist" containerID="c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb" Nov 03 17:02:55.577964 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.577923 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb"} err="failed to get container status \"c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb\": rpc error: code = NotFound desc = could not find container \"c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb\": container with ID starting with c023c81f46f35e80c690bf1b591e7e727adb7db925710782cc31fc8a1d7d45fb not found: ID does not exist" Nov 03 17:02:55.810858 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.810822 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x"] Nov 03 17:02:55.825506 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:55.825430 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/distinct-2-w2-c88d54-kserve-687669d5c8-qns4x"] Nov 03 17:02:56.003809 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.003778 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" path="/var/lib/kubelet/pods/12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44/volumes" Nov 03 17:02:56.702942 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.702920 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:56.775474 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.775436 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-xks-ca\") pod \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " Nov 03 17:02:56.775644 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.775547 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-openshift-ca\") pod \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " Nov 03 17:02:56.775644 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.775632 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b201fc4-3161-44bd-aec1-eea4b7748f5e-kube-api-access-zfckd\" (UniqueName: \"kubernetes.io/projected/3b201fc4-3161-44bd-aec1-eea4b7748f5e-kube-api-access-zfckd\") pod \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\" (UID: \"3b201fc4-3161-44bd-aec1-eea4b7748f5e\") " Nov 03 17:02:56.775965 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.775935 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-openshift-ca" pod "3b201fc4-3161-44bd-aec1-eea4b7748f5e" (UID: "3b201fc4-3161-44bd-aec1-eea4b7748f5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:02:56.777701 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.777679 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-xks-ca" pod "3b201fc4-3161-44bd-aec1-eea4b7748f5e" (UID: "3b201fc4-3161-44bd-aec1-eea4b7748f5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:02:56.777808 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.777721 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b201fc4-3161-44bd-aec1-eea4b7748f5e-kube-api-access-zfckd" pod "3b201fc4-3161-44bd-aec1-eea4b7748f5e" (UID: "3b201fc4-3161-44bd-aec1-eea4b7748f5e"). InnerVolumeSpecName "kube-api-access-zfckd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:02:56.876847 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.876769 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfckd\" (UniqueName: \"kubernetes.io/projected/3b201fc4-3161-44bd-aec1-eea4b7748f5e-kube-api-access-zfckd\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:56.876847 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.876796 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:56.876847 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:56.876805 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b201fc4-3161-44bd-aec1-eea4b7748f5e-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:02:57.581199 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:57.581163 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" event={"ID":"3b201fc4-3161-44bd-aec1-eea4b7748f5e","Type":"ContainerDied","Data":"dbaf5705f7fcd6793c7e87516ee2123fdf8a3732c9444d148ea712686f492dd9"} Nov 03 17:02:57.581199 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:57.581180 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc" Nov 03 17:02:57.581509 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:02:57.581238 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbaf5705f7fcd6793c7e87516ee2123fdf8a3732c9444d148ea712686f492dd9" Nov 03 17:03:08.695813 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:08.695770 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:09.453308 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:09.453273 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:03:10.231643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:10.231613 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:10.462515 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:10.462476 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:10.661850 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:10.661759 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs"] Nov 03 17:03:10.712994 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:10.712959 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c88d54-b62fs"] Nov 03 17:03:11.083687 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:11.083650 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:12.002637 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:12.002596 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92cfa34-6405-44c9-b375-734cf71af873" path="/var/lib/kubelet/pods/d92cfa34-6405-44c9-b375-734cf71af873/volumes" Nov 03 17:03:12.585278 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:03:12.585243 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:03:12.807595 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:12.807556 5946 scope.go:122] "RemoveContainer" containerID="ebdb5f89b4e458623af369acad408bfeebd91e98f8c730c754620250ea37c34b" Nov 03 17:03:16.284256 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:16.284191 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:18.488946 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:18.488897 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:18.862846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:18.862757 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:19.075160 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:19.075125 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:20.809805 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:20.809756 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:21.405996 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:21.405960 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:21.592555 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:21.592521 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:21.800915 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:21.800880 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:24.393737 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:24.393698 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:03:27.550229 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.550172 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-w5sxs"] Nov 03 17:03:27.550658 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.550375 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" podUID="46d86ba8-5de5-42ed-a4b4-7201dc969002" containerName="manager" containerID="cri-o://786cd5a3590f881db6c2e91641bd39b9e650666f0671c6b93d0de5c34cf44427" gracePeriod=30 Nov 03 17:03:27.756393 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.756361 5946 generic.go:356] "Generic (PLEG): container finished" podID="46d86ba8-5de5-42ed-a4b4-7201dc969002" containerID="786cd5a3590f881db6c2e91641bd39b9e650666f0671c6b93d0de5c34cf44427" exitCode=0 Nov 03 17:03:27.756566 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.756445 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" event={"ID":"46d86ba8-5de5-42ed-a4b4-7201dc969002","Type":"ContainerDied","Data":"786cd5a3590f881db6c2e91641bd39b9e650666f0671c6b93d0de5c34cf44427"} Nov 03 17:03:27.802138 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.802082 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 17:03:27.855775 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.855737 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-metrics-certs\") pod \"46d86ba8-5de5-42ed-a4b4-7201dc969002\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " Nov 03 17:03:27.858043 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.858001 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-metrics-certs" pod "46d86ba8-5de5-42ed-a4b4-7201dc969002" (UID: "46d86ba8-5de5-42ed-a4b4-7201dc969002"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:03:27.956385 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.956353 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-webhook-cert\") pod \"46d86ba8-5de5-42ed-a4b4-7201dc969002\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " Nov 03 17:03:27.956579 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.956413 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d86ba8-5de5-42ed-a4b4-7201dc969002-kube-api-access-vrkkn\" (UniqueName: \"kubernetes.io/projected/46d86ba8-5de5-42ed-a4b4-7201dc969002-kube-api-access-vrkkn\") pod \"46d86ba8-5de5-42ed-a4b4-7201dc969002\" (UID: \"46d86ba8-5de5-42ed-a4b4-7201dc969002\") " Nov 03 17:03:27.956748 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.956728 5946 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-metrics-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:03:27.958693 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.958651 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-webhook-cert" pod "46d86ba8-5de5-42ed-a4b4-7201dc969002" (UID: "46d86ba8-5de5-42ed-a4b4-7201dc969002"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:03:27.958837 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:27.958808 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d86ba8-5de5-42ed-a4b4-7201dc969002-kube-api-access-vrkkn" pod "46d86ba8-5de5-42ed-a4b4-7201dc969002" (UID: "46d86ba8-5de5-42ed-a4b4-7201dc969002"). InnerVolumeSpecName "kube-api-access-vrkkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:03:28.057374 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.057284 5946 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d86ba8-5de5-42ed-a4b4-7201dc969002-webhook-cert\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:03:28.057374 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.057320 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrkkn\" (UniqueName: \"kubernetes.io/projected/46d86ba8-5de5-42ed-a4b4-7201dc969002-kube-api-access-vrkkn\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:03:28.348298 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.348222 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:28.763641 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.763610 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" Nov 03 17:03:28.763641 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.763626 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-w5sxs" event={"ID":"46d86ba8-5de5-42ed-a4b4-7201dc969002","Type":"ContainerDied","Data":"4c4c6965a003e69e51cfcf097767a95c7192496dac2f94d8a20fba094918b92a"} Nov 03 17:03:28.764143 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.763716 5946 scope.go:122] "RemoveContainer" containerID="786cd5a3590f881db6c2e91641bd39b9e650666f0671c6b93d0de5c34cf44427" Nov 03 17:03:28.940411 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.940371 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-w5sxs"] Nov 03 17:03:28.991309 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:28.991277 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-w5sxs"] Nov 03 17:03:30.002957 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:30.002921 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d86ba8-5de5-42ed-a4b4-7201dc969002" path="/var/lib/kubelet/pods/46d86ba8-5de5-42ed-a4b4-7201dc969002/volumes" Nov 03 17:03:41.933372 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:41.933336 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:42.584465 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:03:42.584429 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:03:49.066497 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:49.066405 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh"] Nov 03 17:03:49.117076 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:49.117033 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095511ef-5fbvh"] Nov 03 17:03:50.003183 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.003149 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358c779a-f372-4d84-948a-262e032e6dd9" path="/var/lib/kubelet/pods/358c779a-f372-4d84-948a-262e032e6dd9/volumes" Nov 03 17:03:50.632458 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632423 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt"] Nov 03 17:03:50.632905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632803 5946 cpu_manager.go:414] "containerMap: removing container" podUID="46d86ba8-5de5-42ed-a4b4-7201dc969002" containerName="manager" Nov 03 17:03:50.632905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632813 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d86ba8-5de5-42ed-a4b4-7201dc969002" containerName="manager" Nov 03 17:03:50.632905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632824 5946 cpu_manager.go:414] "containerMap: removing container" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" Nov 03 17:03:50.632905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632828 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" Nov 03 17:03:50.632905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632851 5946 cpu_manager.go:414] "containerMap: removing container" podUID="3b201fc4-3161-44bd-aec1-eea4b7748f5e" containerName="revoke-keys" Nov 03 17:03:50.632905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632856 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b201fc4-3161-44bd-aec1-eea4b7748f5e" containerName="revoke-keys" Nov 03 17:03:50.632905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632907 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d86ba8-5de5-42ed-a4b4-7201dc969002" containerName="manager" Nov 03 17:03:50.633181 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632917 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="12dee1e9-3e3f-4bb0-99e0-baf51e1d8e44" containerName="main" Nov 03 17:03:50.633181 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.632928 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b201fc4-3161-44bd-aec1-eea4b7748f5e" containerName="revoke-keys" Nov 03 17:03:50.636325 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.636299 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.699302 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.699270 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mw47n\"" Nov 03 17:03:50.749051 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749015 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2070296-e232-4796-a766-50240d2a2286-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749268 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749059 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749268 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749132 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2070296-e232-4796-a766-50240d2a2286-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749268 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749160 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2070296-e232-4796-a766-50240d2a2286-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749268 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749237 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749268 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749264 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749622 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749299 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5svw6\" (UniqueName: \"kubernetes.io/projected/f2070296-e232-4796-a766-50240d2a2286-kube-api-access-5svw6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749622 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749324 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.749622 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.749348 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.752562 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.752536 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt"] Nov 03 17:03:50.850260 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850198 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2070296-e232-4796-a766-50240d2a2286-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850442 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850267 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850439 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2070296-e232-4796-a766-50240d2a2286-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850525 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850483 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2070296-e232-4796-a766-50240d2a2286-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850568 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850557 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850618 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850587 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850675 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850619 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850675 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850626 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5svw6\" (UniqueName: \"kubernetes.io/projected/f2070296-e232-4796-a766-50240d2a2286-kube-api-access-5svw6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850802 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850672 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850802 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850701 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.850996 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.850974 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.851112 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.851046 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.851166 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.851133 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2070296-e232-4796-a766-50240d2a2286-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.851235 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.851102 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.852843 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.852820 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2070296-e232-4796-a766-50240d2a2286-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.852942 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.852923 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2070296-e232-4796-a766-50240d2a2286-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.911377 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.911314 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2070296-e232-4796-a766-50240d2a2286-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.911482 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.911420 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5svw6\" (UniqueName: \"kubernetes.io/projected/f2070296-e232-4796-a766-50240d2a2286-kube-api-access-5svw6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-kw9pt\" (UID: \"f2070296-e232-4796-a766-50240d2a2286\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:50.948472 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:50.948438 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:51.379548 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:51.379497 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt"] Nov 03 17:03:51.381802 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:03:51.381774 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2070296_e232_4796_a766_50240d2a2286.slice/crio-8802c42d4a3db5ad92b4eaa2aef4f5f7bb00588da5baa55d8cb2a13ea38c3700 WatchSource:0}: Error finding container 8802c42d4a3db5ad92b4eaa2aef4f5f7bb00588da5baa55d8cb2a13ea38c3700: Status 404 returned error can't find the container with id 8802c42d4a3db5ad92b4eaa2aef4f5f7bb00588da5baa55d8cb2a13ea38c3700 Nov 03 17:03:51.383869 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:51.383841 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:03:51.383934 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:51.383906 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:03:51.383977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:51.383933 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:03:51.891990 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:51.891956 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" event={"ID":"f2070296-e232-4796-a766-50240d2a2286","Type":"ContainerStarted","Data":"37d3e2d05e97f14fde8a80c2e8b11a2ce15226a0ec37fd4a9b896c7f100b4df3"} Nov 03 17:03:51.892421 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:51.892007 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" event={"ID":"f2070296-e232-4796-a766-50240d2a2286","Type":"ContainerStarted","Data":"8802c42d4a3db5ad92b4eaa2aef4f5f7bb00588da5baa55d8cb2a13ea38c3700"} Nov 03 17:03:51.948628 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:51.948592 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:03:52.038234 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:52.037418 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podStartSLOduration=2.037394964 podStartE2EDuration="2.037394964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:52.035177385 +0000 UTC m=+3102.156230250" watchObservedRunningTime="2026-11-03 17:03:52.037394964 +0000 UTC m=+3102.158447789" Nov 03 17:03:52.950921 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:52.950889 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 17:03:52.951366 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:52.950951 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 17:03:53.949844 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:53.949806 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 17:03:53.950037 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:53.949866 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 17:03:54.948838 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:54.948798 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 17:03:54.949309 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:54.948857 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 17:03:55.948898 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:55.948860 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 17:03:55.951383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:55.948913 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 17:03:56.948921 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:56.948884 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 17:03:56.949357 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:56.948938 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 17:03:57.948779 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:57.948746 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Nov 03 17:03:57.948961 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:57.948796 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Nov 03 17:03:59.949361 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:59.949314 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:03:59.949783 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:03:59.949374 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:04:00.991397 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:00.991353 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:04:00.991818 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:00.991417 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:04:01.581012 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:01.580977 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:04:01.581346 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:01.581321 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:04:01.582029 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:01.582013 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" Nov 03 17:04:02.065393 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:02.065361 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9"] Nov 03 17:04:02.070083 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:02.070057 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1d4835-wrbw9"] Nov 03 17:04:03.560480 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:03.560443 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:04.003487 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:04.003447 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794fb6a6-ff16-4e5c-8050-52642a53d11c" path="/var/lib/kubelet/pods/794fb6a6-ff16-4e5c-8050-52642a53d11c/volumes" Nov 03 17:04:07.459615 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:07.459560 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:12.060323 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:12.060289 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww"] Nov 03 17:04:12.111080 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:12.111043 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c285dc-m95ww"] Nov 03 17:04:12.584936 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:04:12.584892 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:04:12.839820 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:12.839741 5946 scope.go:122] "RemoveContainer" containerID="b3e763cfc45a8e2d2e0695b9517051d689d296dcd3ab5683182c397fb70e1b55" Nov 03 17:04:12.849178 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:12.849155 5946 scope.go:122] "RemoveContainer" containerID="04ca839fd20cbc9639778cb3b6017963ceaef70c060cbf68fd885071088df07d" Nov 03 17:04:14.002838 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:14.002806 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93917e2-6ca6-48b8-86ba-3d959b0bbba9" path="/var/lib/kubelet/pods/e93917e2-6ca6-48b8-86ba-3d959b0bbba9/volumes" Nov 03 17:04:41.070729 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:41.070694 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2"] Nov 03 17:04:41.126695 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:41.123595 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd71af3-q86t2"] Nov 03 17:04:42.003357 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:42.003307 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4126a9b0-6167-4f41-87e6-676cc3dbb8b2" path="/var/lib/kubelet/pods/4126a9b0-6167-4f41-87e6-676cc3dbb8b2/volumes" Nov 03 17:04:42.316865 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:42.316783 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:42.584950 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:04:42.584855 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:04:56.635261 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:56.635224 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:58.857326 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:04:58.857287 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:05:12.584703 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:05:12.584660 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:05:12.884441 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:12.884346 5946 scope.go:122] "RemoveContainer" containerID="be5066902d6f998dc7c0525a1551e1598634e9d23dc8f14b54b7ca938a3ee81b" Nov 03 17:05:12.893317 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:12.893295 5946 scope.go:122] "RemoveContainer" containerID="2ef0249e7071ac7a6d54680bec190e24cf48d320aa29e0e7bd9b8ecf776b6ed2" Nov 03 17:05:12.901663 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:12.901645 5946 scope.go:122] "RemoveContainer" containerID="ba7e1f80cbb39d4be57419f7cc3d2b95a2c698f67da23926fc54c6b3e3a4c26e" Nov 03 17:05:12.909649 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:12.909632 5946 scope.go:122] "RemoveContainer" containerID="c22b30ce6e4c29ebd110961082a5a78e2db7be2d32fb29b793e8b13ad94bb1d9" Nov 03 17:05:20.197884 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:20.197844 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:05:21.383826 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.383789 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg"] Nov 03 17:05:21.388938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.388916 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490288 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490254 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490478 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490299 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490478 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490327 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490478 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490411 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490478 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490455 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490478 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490475 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490745 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490584 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tz5\" (UniqueName: \"kubernetes.io/projected/72b9663c-c665-4a5a-a2c7-c564bd2696b0-kube-api-access-s4tz5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490745 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490705 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.490829 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.490747 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592322 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592285 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592337 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592382 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592430 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tz5\" (UniqueName: \"kubernetes.io/projected/72b9663c-c665-4a5a-a2c7-c564bd2696b0-kube-api-access-s4tz5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592501 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592526 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592742 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592589 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592742 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592612 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592742 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592637 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.592870 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.592845 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.593065 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.593038 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.593144 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.593109 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.593234 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.593153 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.593345 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.593323 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.595242 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.595220 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.595712 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.595694 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.604645 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.604618 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg"] Nov 03 17:05:21.653590 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.653514 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b9663c-c665-4a5a-a2c7-c564bd2696b0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.653739 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.653612 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tz5\" (UniqueName: \"kubernetes.io/projected/72b9663c-c665-4a5a-a2c7-c564bd2696b0-kube-api-access-s4tz5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-w8lvg\" (UID: \"72b9663c-c665-4a5a-a2c7-c564bd2696b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:21.702000 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:21.701965 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:22.133987 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.133956 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg"] Nov 03 17:05:22.136740 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:05:22.136710 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b9663c_c665_4a5a_a2c7_c564bd2696b0.slice/crio-86973e3418bf741f0a7bb8ee1f38ed2b0e6cc99ca76311a11696c4c4aab20f43 WatchSource:0}: Error finding container 86973e3418bf741f0a7bb8ee1f38ed2b0e6cc99ca76311a11696c4c4aab20f43: Status 404 returned error can't find the container with id 86973e3418bf741f0a7bb8ee1f38ed2b0e6cc99ca76311a11696c4c4aab20f43 Nov 03 17:05:22.138874 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.138845 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:05:22.138966 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.138904 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:05:22.138966 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.138934 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:05:22.427888 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.427798 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" event={"ID":"72b9663c-c665-4a5a-a2c7-c564bd2696b0","Type":"ContainerStarted","Data":"2df9c0a386390f576b6c6724992518e352e7166904cb2047b9cc8a1dfedc1440"} Nov 03 17:05:22.427888 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.427849 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" event={"ID":"72b9663c-c665-4a5a-a2c7-c564bd2696b0","Type":"ContainerStarted","Data":"86973e3418bf741f0a7bb8ee1f38ed2b0e6cc99ca76311a11696c4c4aab20f43"} Nov 03 17:05:22.556191 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.556133 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podStartSLOduration=1.5561163009999999 podStartE2EDuration="1.556116301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:22.554114811 +0000 UTC m=+3192.675167633" watchObservedRunningTime="2026-11-03 17:05:22.556116301 +0000 UTC m=+3192.677169115" Nov 03 17:05:22.702394 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:22.702315 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:23.704222 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:23.704183 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 03 17:05:23.704646 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:23.704252 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 03 17:05:24.703060 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:24.703018 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 03 17:05:24.703274 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:24.703082 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 03 17:05:25.702405 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:25.702367 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 03 17:05:25.702909 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:25.702433 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 03 17:05:26.702745 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:26.702703 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 03 17:05:26.703159 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:26.702773 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 03 17:05:27.702613 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:27.702578 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 03 17:05:27.702817 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:27.702636 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 03 17:05:28.703060 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:28.703023 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 03 17:05:28.703478 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:28.703106 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 03 17:05:30.325357 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:30.325323 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:30.325873 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:30.325745 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:30.326691 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:30.326668 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" Nov 03 17:05:34.985545 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:34.985510 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:05:36.510755 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.510723 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f"] Nov 03 17:05:36.514701 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.514190 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.612226 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.612179 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f"] Nov 03 17:05:36.668263 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.668201 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d902919-9525-449c-9b32-c58e0fde1618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.668565 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.668533 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.668757 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.668721 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.668955 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.668919 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.669301 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.669271 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.669501 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.669472 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.669661 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.669643 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d902919-9525-449c-9b32-c58e0fde1618-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.669819 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.669802 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8gs\" (UniqueName: \"kubernetes.io/projected/1d902919-9525-449c-9b32-c58e0fde1618-kube-api-access-pc8gs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.669975 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.669959 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d902919-9525-449c-9b32-c58e0fde1618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.771831 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.771722 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d902919-9525-449c-9b32-c58e0fde1618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.772146 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.772101 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.772393 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.772358 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.772562 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.772546 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.772774 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.772757 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.772939 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.772920 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.773103 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.773086 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d902919-9525-449c-9b32-c58e0fde1618-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.773277 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.773247 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8gs\" (UniqueName: \"kubernetes.io/projected/1d902919-9525-449c-9b32-c58e0fde1618-kube-api-access-pc8gs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.773436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.773418 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d902919-9525-449c-9b32-c58e0fde1618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.774741 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.774721 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d902919-9525-449c-9b32-c58e0fde1618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.776608 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.776584 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.778559 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.777690 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.778713 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.777926 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.778836 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.778493 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.780835 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.780812 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d902919-9525-449c-9b32-c58e0fde1618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.781139 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.781119 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d902919-9525-449c-9b32-c58e0fde1618-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.838909 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.838734 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8gs\" (UniqueName: \"kubernetes.io/projected/1d902919-9525-449c-9b32-c58e0fde1618-kube-api-access-pc8gs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:36.841092 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:36.841036 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d902919-9525-449c-9b32-c58e0fde1618-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcq7f\" (UID: \"1d902919-9525-449c-9b32-c58e0fde1618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:37.139356 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:37.139242 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:37.605521 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:37.605479 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f"] Nov 03 17:05:37.611905 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:05:37.611874 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d902919_9525_449c_9b32_c58e0fde1618.slice/crio-d2472506b4e86f43f486a51fd26359db4639322e90dc343cde542ce3b1765950 WatchSource:0}: Error finding container d2472506b4e86f43f486a51fd26359db4639322e90dc343cde542ce3b1765950: Status 404 returned error can't find the container with id d2472506b4e86f43f486a51fd26359db4639322e90dc343cde542ce3b1765950 Nov 03 17:05:37.614642 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:37.614598 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:05:37.614749 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:37.614692 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:05:37.614749 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:37.614735 5946 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:05:38.542278 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:38.542241 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" event={"ID":"1d902919-9525-449c-9b32-c58e0fde1618","Type":"ContainerStarted","Data":"4204defa36e5002c10f897626e7b37cde5a13671c87765a451f81a09d16576e6"} Nov 03 17:05:38.542512 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:38.542298 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" event={"ID":"1d902919-9525-449c-9b32-c58e0fde1618","Type":"ContainerStarted","Data":"d2472506b4e86f43f486a51fd26359db4639322e90dc343cde542ce3b1765950"} Nov 03 17:05:39.142865 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:39.140164 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:39.142865 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:39.142271 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:05:39.142865 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:39.142455 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:05:40.140925 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:40.140892 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:05:40.141127 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:40.140949 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:05:40.385039 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:40.385008 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:41.140612 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:41.140575 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:05:41.140839 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:41.140641 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:05:42.070298 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:42.070266 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:42.140484 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:42.140446 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:05:42.140710 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:42.140507 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:05:42.586271 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:05:42.586229 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:05:42.590196 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:42.590168 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:43.140713 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:43.140678 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:05:43.141106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:43.140760 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:05:44.140770 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:44.140683 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:05:44.140770 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:44.140738 5946 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:05:45.782682 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:45.782650 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:45.783183 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:45.783012 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:45.784019 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:45.783996 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" Nov 03 17:05:45.957479 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:45.957415 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podStartSLOduration=9.957402225 podStartE2EDuration="9.957402225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:38.735481139 +0000 UTC m=+3208.856533965" watchObservedRunningTime="2026-11-03 17:05:45.957402225 +0000 UTC m=+3216.078455045" Nov 03 17:05:52.197802 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:52.197763 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:52.403353 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:52.403317 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:55.368457 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:05:55.368416 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:06:12.499499 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:12.499464 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:06:12.585195 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:06:12.585159 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:06:12.944872 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:12.944765 5946 scope.go:122] "RemoveContainer" containerID="d39b569404955135d858dbcebbb69025cff30b69bf242368d2796ce938a978fd" Nov 03 17:06:12.958939 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:12.958912 5946 scope.go:122] "RemoveContainer" containerID="2c52acc2dbda653d453702b33c5041cb65cdfafbc95e0556386a9c432db843e9" Nov 03 17:06:12.970504 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:12.970479 5946 scope.go:122] "RemoveContainer" containerID="64462b4d5c99299c5a74ff38757b970778d53741d48407c9f5628814533eaa08" Nov 03 17:06:12.982481 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:12.982446 5946 scope.go:122] "RemoveContainer" containerID="bb72dccdfd607bc43764d457e6c8fe3709f50d4ddcde4a5dc4522b7a5271fbfb" Nov 03 17:06:12.993359 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:12.993326 5946 scope.go:122] "RemoveContainer" containerID="57cea5304faa3b5d2639ce37ce23011af68579389fdd60544a377173a6787125" Nov 03 17:06:16.061292 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:16.061255 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf"] Nov 03 17:06:16.112294 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:16.112257 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx"] Nov 03 17:06:16.113869 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:16.113841 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26326065-x4jtf"] Nov 03 17:06:16.115140 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:16.115118 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0cb1e6-zrjwx"] Nov 03 17:06:16.253779 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:16.253740 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:17.199060 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:17.199028 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:18.003352 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:18.003318 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946fd444-b286-4bba-aa60-c758f8789520" path="/var/lib/kubelet/pods/946fd444-b286-4bba-aa60-c758f8789520/volumes" Nov 03 17:06:18.003733 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:18.003719 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77932d8-246b-45ef-9486-0b1af87117ff" path="/var/lib/kubelet/pods/b77932d8-246b-45ef-9486-0b1af87117ff/volumes" Nov 03 17:06:25.100812 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:25.100776 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:42.585012 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:06:42.584970 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:06:42.943845 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:42.943773 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:54.662738 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:06:54.662652 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:07:05.470531 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:05.470495 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:07:10.060589 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:10.060558 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr"] Nov 03 17:07:10.113814 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:10.113775 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d48023-8rlmr"] Nov 03 17:07:10.546715 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:10.546684 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:07:10.561853 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:10.561829 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:07:12.003076 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:12.003039 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a6c1d2-abc9-491b-9317-edba35b8ab4d" path="/var/lib/kubelet/pods/09a6c1d2-abc9-491b-9317-edba35b8ab4d/volumes" Nov 03 17:07:12.590139 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:07:12.590095 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:07:13.032599 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:13.032558 5946 scope.go:122] "RemoveContainer" containerID="a45caf01aca113cadbe0202a782cae0a9432463b7c87be2ebb640adf4db09765" Nov 03 17:07:13.042981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:13.042939 5946 scope.go:122] "RemoveContainer" containerID="5a9a1cbd38efa5efdf9300246f1ffe4152741ba55922fc7adbc2fe82ad0dde7c" Nov 03 17:07:13.054064 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:13.053849 5946 scope.go:122] "RemoveContainer" containerID="056c7be22dad2b1e5564e1f05474c45bcf71eab8ebd622d9478e9647526297cd" Nov 03 17:07:13.065271 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:13.065245 5946 scope.go:122] "RemoveContainer" containerID="76723dcdbc94e35b706a5551e9c5669d05d05409679b6693cd4a61041541d9cb" Nov 03 17:07:13.075496 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:13.075472 5946 scope.go:122] "RemoveContainer" containerID="eff34a5b02722916b94ee498573c46eb666d4795795da082eaafcaa30ba410d6" Nov 03 17:07:18.068455 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:18.068417 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:07:40.188242 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:40.188192 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:07:42.585683 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:07:42.585633 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:07:43.302534 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:43.302498 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:43.495981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:43.495950 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:47.626233 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:47.623372 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:07:47.626233 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:47.623463 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:07:49.788942 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:49.788895 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:07:57.060598 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:57.060562 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc"] Nov 03 17:07:57.111453 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:57.111422 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd062fd-vm9zc"] Nov 03 17:07:58.003199 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:07:58.003166 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b201fc4-3161-44bd-aec1-eea4b7748f5e" path="/var/lib/kubelet/pods/3b201fc4-3161-44bd-aec1-eea4b7748f5e/volumes" Nov 03 17:08:08.381350 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:08.381311 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:08:12.585756 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:08:12.585715 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:08:13.126446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:13.126410 5946 scope.go:122] "RemoveContainer" containerID="3e2cc0a2b5b7b53cd0c893bf69bdfc07bea28f4b6ea852dd25cec81ad7a1a7b0" Nov 03 17:08:13.135106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:13.135087 5946 scope.go:122] "RemoveContainer" containerID="564b19039fccd9156ab4f605bbdd8f59bbe99c6f7e0e4e6add0f4a254c4a1818" Nov 03 17:08:13.142974 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:13.142952 5946 scope.go:122] "RemoveContainer" containerID="10e6440f2b81ec4a042a2265389395c77a297b0fdedd8451bd87d0ab7691398f" Nov 03 17:08:13.150465 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:13.150447 5946 scope.go:122] "RemoveContainer" containerID="0abb846d49841e092f1f4245390a78c6c6fed1a83770b0b33e3c37f00ab25444" Nov 03 17:08:20.459325 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:20.459287 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:08:40.650494 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:40.650461 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:08:42.334237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:42.334058 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:42.607173 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:08:42.607088 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:08:43.604245 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:08:43.604199 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:02.692655 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:02.692608 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:09:12.585248 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:09:12.585190 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:09:13.183102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:13.183068 5946 scope.go:122] "RemoveContainer" containerID="3d481c6126093f5143899de52285801771d0a0356518343a266c859d8d08f85b" Nov 03 17:09:13.191580 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:13.191559 5946 scope.go:122] "RemoveContainer" containerID="db0ed2e16bc08374ea9d634663a1bd95d258ceffd8cb8f61cfb01ea4e55e05b5" Nov 03 17:09:15.745532 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:15.745495 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:09:26.211685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:26.211651 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:09:42.584888 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:09:42.584844 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:09:43.937970 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:43.937938 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:46.379559 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:46.379480 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:09:56.291020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:09:56.290983 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:10:03.378754 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:10:03.378719 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:11.794569 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:10:11.794530 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:10:12.589551 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:10:12.589506 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:10:34.871337 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:10:34.871294 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:10:42.584350 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:10:42.584311 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:10:44.289569 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:10:44.289533 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:45.686086 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:10:45.686049 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:11:06.368780 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:06.368740 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:11:12.587297 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:11:12.587253 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:11:16.658158 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:16.658109 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:11:33.545068 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:33.545022 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:34.201413 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:34.201364 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:34.400668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:34.400632 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:42.585409 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:11:42.585367 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:11:44.621966 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:44.621929 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:48.377370 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:48.377336 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:48.607424 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:48.607383 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:48.815612 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:48.815580 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:51.622493 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:51.622458 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:51.833699 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:51.833668 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:55.288528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:55.288494 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:55.489996 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:55.489963 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:55.691985 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:11:55.691903 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:00.655701 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:00.655662 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:03.455632 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:03.455590 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:12:05.941760 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:05.941705 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:10.583839 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:10.583811 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:12:10.597464 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:10.597440 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:12:11.234081 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:11.234031 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:12.585085 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:12:12.585048 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:12:13.133381 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:13.133345 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:13.744510 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:13.744475 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:13.939281 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:13.939240 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:16.697303 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.697270 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4"] Nov 03 17:12:16.700904 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.700889 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.806300 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.806265 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-433fd6\"/\"kube-root-ca.crt\"" Nov 03 17:12:16.806300 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.806265 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-433fd6\"/\"openshift-service-ca.crt\"" Nov 03 17:12:16.806300 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.806278 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-433fd6\"/\"default-dockercfg-v8mj8\"" Nov 03 17:12:16.806573 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.806361 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-433fd6\"/\"simulator-single-main-433fd6-kserve-self-signed-certs\"" Nov 03 17:12:16.808849 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.808829 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4"] Nov 03 17:12:16.825931 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.825902 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277b8e35-9662-474c-988d-f226d40879e1-tls-certs\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.826072 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.825945 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-model-cache\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.826072 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.826053 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-tmp-dir\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.826150 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.826077 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-home\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.826150 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.826114 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q52bf\" (UniqueName: \"kubernetes.io/projected/277b8e35-9662-474c-988d-f226d40879e1-kube-api-access-q52bf\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.826272 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.826186 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-dshm\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.842597 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.842548 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:16.927260 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927194 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277b8e35-9662-474c-988d-f226d40879e1-tls-certs\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927281 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-model-cache\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927328 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-tmp-dir\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927348 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-home\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927386 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52bf\" (UniqueName: \"kubernetes.io/projected/277b8e35-9662-474c-988d-f226d40879e1-kube-api-access-q52bf\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927402 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-dshm\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927775 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927747 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-model-cache\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927878 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927786 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-tmp-dir\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.927878 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.927825 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-home\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.929758 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.929728 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-dshm\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.929874 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.929856 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277b8e35-9662-474c-988d-f226d40879e1-tls-certs\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:16.990843 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:16.990811 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52bf\" (UniqueName: \"kubernetes.io/projected/277b8e35-9662-474c-988d-f226d40879e1-kube-api-access-q52bf\") pod \"simulator-single-main-433fd6-kserve-64c94c575b-clcp4\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:17.013924 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:17.013887 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:17.028997 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:17.028970 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:17.402084 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:17.402056 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4"] Nov 03 17:12:17.403787 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:12:17.403753 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod277b8e35_9662_474c_988d_f226d40879e1.slice/crio-b45226edb8f0f9713f4d005e9f3f67619067f74aeef622732d1f75ca4567c317 WatchSource:0}: Error finding container b45226edb8f0f9713f4d005e9f3f67619067f74aeef622732d1f75ca4567c317: Status 404 returned error can't find the container with id b45226edb8f0f9713f4d005e9f3f67619067f74aeef622732d1f75ca4567c317 Nov 03 17:12:17.705685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:17.705597 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" event={"ID":"277b8e35-9662-474c-988d-f226d40879e1","Type":"ContainerStarted","Data":"81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9"} Nov 03 17:12:17.705685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:17.705639 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" event={"ID":"277b8e35-9662-474c-988d-f226d40879e1","Type":"ContainerStarted","Data":"b45226edb8f0f9713f4d005e9f3f67619067f74aeef622732d1f75ca4567c317"} Nov 03 17:12:17.826457 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:17.826394 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" podStartSLOduration=1.826374538 podStartE2EDuration="1.826374538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:17.825509816 +0000 UTC m=+3607.946562640" watchObservedRunningTime="2026-11-03 17:12:17.826374538 +0000 UTC m=+3607.947427361" Nov 03 17:12:18.014220 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:18.014188 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:21.371338 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:21.371300 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:12:22.082436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:22.082398 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:27.014383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:27.014345 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:27.026922 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:27.026893 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:27.287049 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:27.286938 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:27.765817 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:27.765775 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:12:32.442644 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:32.442613 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:37.747333 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:37.747298 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:42.585658 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:12:42.585622 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:12:42.944369 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:42.944263 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:45.021928 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:45.021844 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:48.315441 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:48.315404 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:52.281990 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:52.281945 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:12:53.471182 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:53.471143 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:58.745621 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:12:58.745587 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:03.873924 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:03.873882 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:09.195079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:09.195046 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:12.584661 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:13:12.584623 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:13:14.350825 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:14.350789 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:16.507527 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.507490 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb"] Nov 03 17:13:16.511020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.511003 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.593520 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.593484 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.593688 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.593527 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblpl\" (UniqueName: \"kubernetes.io/projected/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-kube-api-access-tblpl\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.593688 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.593642 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.616321 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.616284 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:13:16.616321 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.616284 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:13:16.616519 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.616298 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:13:16.616519 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.616327 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:13:16.617846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.617827 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb"] Nov 03 17:13:16.695095 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.695059 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.695323 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.695169 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.695323 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.695201 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblpl\" (UniqueName: \"kubernetes.io/projected/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-kube-api-access-tblpl\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.695726 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.695701 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.697705 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.697686 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.757021 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.756986 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblpl\" (UniqueName: \"kubernetes.io/projected/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-kube-api-access-tblpl\") pod \"maas-api-revoke-sub-f68d2fd7-4m2nb\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:16.823970 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:16.823887 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:17.203761 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:17.203734 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb"] Nov 03 17:13:17.206343 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:13:17.206313 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3d7ee5_43d0_450a_89a2_acbaf4accfd8.slice/crio-9fec35d4d1c79bd880db2902fb71235efd544a0480d95b0f83dc915a191b30d8 WatchSource:0}: Error finding container 9fec35d4d1c79bd880db2902fb71235efd544a0480d95b0f83dc915a191b30d8: Status 404 returned error can't find the container with id 9fec35d4d1c79bd880db2902fb71235efd544a0480d95b0f83dc915a191b30d8 Nov 03 17:13:17.995000 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:17.994963 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" event={"ID":"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8","Type":"ContainerStarted","Data":"a4ea325f76c21524878263c6015693a849f65c3d7718b949bf8bb686dff45dfd"} Nov 03 17:13:17.995492 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:17.995017 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" event={"ID":"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8","Type":"ContainerStarted","Data":"9fec35d4d1c79bd880db2902fb71235efd544a0480d95b0f83dc915a191b30d8"} Nov 03 17:13:18.114751 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:18.114698 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" podStartSLOduration=2.114684188 podStartE2EDuration="2.114684188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:13:18.114394146 +0000 UTC m=+3668.235446969" watchObservedRunningTime="2026-11-03 17:13:18.114684188 +0000 UTC m=+3668.235737011" Nov 03 17:13:19.001684 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:19.001647 5946 generic.go:356] "Generic (PLEG): container finished" podID="ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" containerID="a4ea325f76c21524878263c6015693a849f65c3d7718b949bf8bb686dff45dfd" exitCode=0 Nov 03 17:13:19.002136 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:19.001736 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" event={"ID":"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8","Type":"ContainerDied","Data":"a4ea325f76c21524878263c6015693a849f65c3d7718b949bf8bb686dff45dfd"} Nov 03 17:13:19.569421 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:19.569340 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:20.155923 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.155887 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:20.233577 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.233541 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-kube-api-access-tblpl\" (UniqueName: \"kubernetes.io/projected/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-kube-api-access-tblpl\") pod \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " Nov 03 17:13:20.233760 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.233595 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-xks-ca\") pod \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " Nov 03 17:13:20.233760 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.233701 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-openshift-ca\") pod \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\" (UID: \"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8\") " Nov 03 17:13:20.234068 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.234042 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-openshift-ca" pod "ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" (UID: "ca3d7ee5-43d0-450a-89a2-acbaf4accfd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:20.235892 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.235867 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-xks-ca" pod "ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" (UID: "ca3d7ee5-43d0-450a-89a2-acbaf4accfd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:13:20.235979 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.235871 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-kube-api-access-tblpl" pod "ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" (UID: "ca3d7ee5-43d0-450a-89a2-acbaf4accfd8"). InnerVolumeSpecName "kube-api-access-tblpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:20.334486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.334408 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:20.334486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.334436 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblpl\" (UniqueName: \"kubernetes.io/projected/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-kube-api-access-tblpl\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:20.334486 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:20.334447 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:21.011585 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:21.011554 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" Nov 03 17:13:21.011585 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:21.011572 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb" event={"ID":"ca3d7ee5-43d0-450a-89a2-acbaf4accfd8","Type":"ContainerDied","Data":"9fec35d4d1c79bd880db2902fb71235efd544a0480d95b0f83dc915a191b30d8"} Nov 03 17:13:21.011859 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:21.011614 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fec35d4d1c79bd880db2902fb71235efd544a0480d95b0f83dc915a191b30d8" Nov 03 17:13:24.754452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:24.754423 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:28.320460 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.320421 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:28.753424 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.753393 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j"] Nov 03 17:13:28.753811 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.753796 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" containerName="revoke-keys" Nov 03 17:13:28.753864 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.753813 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" containerName="revoke-keys" Nov 03 17:13:28.753909 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.753905 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" containerName="revoke-keys" Nov 03 17:13:28.757428 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.757407 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:28.815811 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.815780 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:13:28.873667 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.866613 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:13:28.873667 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.866749 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:13:28.873667 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.866760 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:13:28.873667 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.870300 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j"] Nov 03 17:13:28.922592 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.922560 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:28.922784 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.922610 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:28.922784 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:28.922758 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdkz4\" (UniqueName: \"kubernetes.io/projected/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-kube-api-access-jdkz4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.023995 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.023901 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.023995 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.023956 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.024266 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.024021 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdkz4\" (UniqueName: \"kubernetes.io/projected/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-kube-api-access-jdkz4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.024722 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.024695 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.026480 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.026460 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.085702 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.085669 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdkz4\" (UniqueName: \"kubernetes.io/projected/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-kube-api-access-jdkz4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.371650 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.371568 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:29.748452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:29.748423 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j"] Nov 03 17:13:29.750550 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:13:29.750521 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ffc9e8_bcf1_4f3e_9cc0_bfe2269f345f.slice/crio-ccbbb4af1f6dd22a9ddd7528bc62cb18a262ba579624666e7e2f71053d6fa743 WatchSource:0}: Error finding container ccbbb4af1f6dd22a9ddd7528bc62cb18a262ba579624666e7e2f71053d6fa743: Status 404 returned error can't find the container with id ccbbb4af1f6dd22a9ddd7528bc62cb18a262ba579624666e7e2f71053d6fa743 Nov 03 17:13:30.052906 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:30.052874 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" event={"ID":"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f","Type":"ContainerStarted","Data":"14d21304faeb49e76d3948a6f188b7d6c7388eb9f89e39df868766e77d96436f"} Nov 03 17:13:30.053046 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:30.052933 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" event={"ID":"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f","Type":"ContainerStarted","Data":"ccbbb4af1f6dd22a9ddd7528bc62cb18a262ba579624666e7e2f71053d6fa743"} Nov 03 17:13:30.139101 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:30.139070 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:30.172920 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:30.172860 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" podStartSLOduration=2.17284637 podStartE2EDuration="2.17284637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:13:30.17214598 +0000 UTC m=+3680.293198804" watchObservedRunningTime="2026-11-03 17:13:30.17284637 +0000 UTC m=+3680.293899190" Nov 03 17:13:30.633872 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:13:30.633845 5946 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ffc9e8_bcf1_4f3e_9cc0_bfe2269f345f.slice/crio-14d21304faeb49e76d3948a6f188b7d6c7388eb9f89e39df868766e77d96436f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:13:31.058882 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:31.058844 5946 generic.go:356] "Generic (PLEG): container finished" podID="e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" containerID="14d21304faeb49e76d3948a6f188b7d6c7388eb9f89e39df868766e77d96436f" exitCode=0 Nov 03 17:13:31.059067 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:31.058887 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" event={"ID":"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f","Type":"ContainerDied","Data":"14d21304faeb49e76d3948a6f188b7d6c7388eb9f89e39df868766e77d96436f"} Nov 03 17:13:31.283444 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:31.283402 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:13:32.203831 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.203808 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:32.356319 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.356199 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-kube-api-access-jdkz4\" (UniqueName: \"kubernetes.io/projected/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-kube-api-access-jdkz4\") pod \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " Nov 03 17:13:32.356319 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.356313 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-xks-ca\") pod \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " Nov 03 17:13:32.356541 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.356377 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-openshift-ca\") pod \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\" (UID: \"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f\") " Nov 03 17:13:32.356802 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.356767 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-openshift-ca" pod "e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" (UID: "e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:32.358925 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.358896 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-xks-ca" pod "e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" (UID: "e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:13:32.358925 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.358914 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-kube-api-access-jdkz4" pod "e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" (UID: "e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f"). InnerVolumeSpecName "kube-api-access-jdkz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:32.457841 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.457803 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:32.457841 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.457837 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:32.457841 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:32.457847 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdkz4\" (UniqueName: \"kubernetes.io/projected/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f-kube-api-access-jdkz4\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:33.070162 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:33.070129 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" Nov 03 17:13:33.070162 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:33.070136 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j" event={"ID":"e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f","Type":"ContainerDied","Data":"ccbbb4af1f6dd22a9ddd7528bc62cb18a262ba579624666e7e2f71053d6fa743"} Nov 03 17:13:33.070405 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:33.070188 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccbbb4af1f6dd22a9ddd7528bc62cb18a262ba579624666e7e2f71053d6fa743" Nov 03 17:13:33.244176 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:33.244140 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4"] Nov 03 17:13:33.387258 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:33.387160 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:35.639890 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:35.639858 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:38.159548 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:38.159515 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:40.163729 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:40.163695 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:40.599918 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:40.599886 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:40.794463 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:40.794424 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:42.585083 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:13:42.585043 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:13:44.497036 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:44.496988 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:44.791396 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:44.791320 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:44.895373 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:44.895342 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:13:45.095557 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:45.095473 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j"] Nov 03 17:13:45.146339 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:45.146305 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-433fd6-c7t4j"] Nov 03 17:13:45.312151 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:45.312098 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:45.576843 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:45.576813 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:46.003026 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:46.002982 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" path="/var/lib/kubelet/pods/e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f/volumes" Nov 03 17:13:48.264835 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.264783 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" podUID="277b8e35-9662-474c-988d-f226d40879e1" containerName="main" containerID="cri-o://81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9" gracePeriod=45 Nov 03 17:13:48.528681 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.528628 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:13:48.609267 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609232 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-tmp-dir\") pod \"277b8e35-9662-474c-988d-f226d40879e1\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " Nov 03 17:13:48.609461 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609291 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277b8e35-9662-474c-988d-f226d40879e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/277b8e35-9662-474c-988d-f226d40879e1-tls-certs\") pod \"277b8e35-9662-474c-988d-f226d40879e1\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " Nov 03 17:13:48.609461 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609324 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-home\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-home\") pod \"277b8e35-9662-474c-988d-f226d40879e1\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " Nov 03 17:13:48.609461 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609357 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-dshm\") pod \"277b8e35-9662-474c-988d-f226d40879e1\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " Nov 03 17:13:48.609643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609490 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-model-cache\") pod \"277b8e35-9662-474c-988d-f226d40879e1\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " Nov 03 17:13:48.609643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609501 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-tmp-dir" pod "277b8e35-9662-474c-988d-f226d40879e1" (UID: "277b8e35-9662-474c-988d-f226d40879e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:13:48.609643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609541 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277b8e35-9662-474c-988d-f226d40879e1-kube-api-access-q52bf\" (UniqueName: \"kubernetes.io/projected/277b8e35-9662-474c-988d-f226d40879e1-kube-api-access-q52bf\") pod \"277b8e35-9662-474c-988d-f226d40879e1\" (UID: \"277b8e35-9662-474c-988d-f226d40879e1\") " Nov 03 17:13:48.609643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609609 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-home" pod "277b8e35-9662-474c-988d-f226d40879e1" (UID: "277b8e35-9662-474c-988d-f226d40879e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:13:48.609812 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609714 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-model-cache" pod "277b8e35-9662-474c-988d-f226d40879e1" (UID: "277b8e35-9662-474c-988d-f226d40879e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:13:48.609932 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609914 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:48.609981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609942 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:48.609981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.609957 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:48.611779 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.611747 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-dshm" pod "277b8e35-9662-474c-988d-f226d40879e1" (UID: "277b8e35-9662-474c-988d-f226d40879e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:13:48.611779 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.611769 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277b8e35-9662-474c-988d-f226d40879e1-kube-api-access-q52bf" pod "277b8e35-9662-474c-988d-f226d40879e1" (UID: "277b8e35-9662-474c-988d-f226d40879e1"). InnerVolumeSpecName "kube-api-access-q52bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:48.611954 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.611807 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277b8e35-9662-474c-988d-f226d40879e1-tls-certs" pod "277b8e35-9662-474c-988d-f226d40879e1" (UID: "277b8e35-9662-474c-988d-f226d40879e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:13:48.711158 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.711124 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277b8e35-9662-474c-988d-f226d40879e1-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:48.711158 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.711154 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277b8e35-9662-474c-988d-f226d40879e1-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:48.711396 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:48.711166 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q52bf\" (UniqueName: \"kubernetes.io/projected/277b8e35-9662-474c-988d-f226d40879e1-kube-api-access-q52bf\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:13:49.150359 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.150324 5946 generic.go:356] "Generic (PLEG): container finished" podID="277b8e35-9662-474c-988d-f226d40879e1" containerID="81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9" exitCode=0 Nov 03 17:13:49.150554 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.150381 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" Nov 03 17:13:49.150554 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.150413 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" event={"ID":"277b8e35-9662-474c-988d-f226d40879e1","Type":"ContainerDied","Data":"81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9"} Nov 03 17:13:49.150554 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.150461 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4" event={"ID":"277b8e35-9662-474c-988d-f226d40879e1","Type":"ContainerDied","Data":"b45226edb8f0f9713f4d005e9f3f67619067f74aeef622732d1f75ca4567c317"} Nov 03 17:13:49.150554 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.150479 5946 scope.go:122] "RemoveContainer" containerID="81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9" Nov 03 17:13:49.160499 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.160467 5946 scope.go:122] "RemoveContainer" containerID="81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9" Nov 03 17:13:49.160747 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:13:49.160726 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9\": container with ID starting with 81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9 not found: ID does not exist" containerID="81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9" Nov 03 17:13:49.160820 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.160760 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9"} err="failed to get container status \"81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9\": rpc error: code = NotFound desc = could not find container \"81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9\": container with ID starting with 81a84845ceb3e5c2c578aab79af06c021f19b66689cbd250104a16648342aaa9 not found: ID does not exist" Nov 03 17:13:49.321909 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.321873 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4"] Nov 03 17:13:49.372783 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:49.372749 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-433fd6/simulator-single-main-433fd6-kserve-64c94c575b-clcp4"] Nov 03 17:13:50.002504 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:50.002464 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277b8e35-9662-474c-988d-f226d40879e1" path="/var/lib/kubelet/pods/277b8e35-9662-474c-988d-f226d40879e1/volumes" Nov 03 17:13:52.548575 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:52.548543 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:52.739643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:52.739612 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:52.951750 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:13:52.951598 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:09.068134 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:14:09.068093 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:14:09.574225 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:14:09.574161 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:11.920378 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:14:11.920346 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:12.585149 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:14:12.585112 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:14:18.580778 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:14:18.580744 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:19.451438 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:14:19.451399 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:14:42.584782 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:14:42.584737 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:14:45.681467 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:14:45.681430 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:11.540754 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:15:11.540708 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:12.585414 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:15:12.585375 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:15:42.586040 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:15:42.585995 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:15:46.021197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:15:46.021101 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:00.350584 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:00.350548 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:03.060245 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:03.060193 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:16:12.584896 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:16:12.584853 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:16:13.750447 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:13.750409 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:20.280737 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:20.280696 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:16:29.489735 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:29.489700 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:16:42.584831 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:16:42.584789 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:16:46.371129 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:46.371038 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:49.171426 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:49.171373 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:49.393391 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:49.393356 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:16:49.911038 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:49.910976 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:50.139081 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:50.139032 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:53.879544 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:53.879514 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:54.076822 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:54.076751 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:57.062522 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:57.062481 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:57.256916 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:57.256885 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:57.691027 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:57.690994 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:57.878604 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:16:57.878537 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:00.587668 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:00.587623 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:00.790373 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:00.790329 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:00.977279 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:00.977171 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:05.961864 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:05.961827 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:08.695574 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:08.695531 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:17:10.619417 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:10.619392 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:17:10.633709 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:10.633684 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:17:11.260115 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:11.260078 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:12.585078 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:17:12.585038 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:17:16.363794 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:16.363753 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:18.616919 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:18.616886 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:19.162700 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:19.162663 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:19.368950 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:19.368909 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:21.947637 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:21.947590 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:22.527719 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.527687 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd"] Nov 03 17:17:22.528323 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.528305 5946 cpu_manager.go:414] "containerMap: removing container" podUID="e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" containerName="revoke-keys" Nov 03 17:17:22.528382 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.528326 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" containerName="revoke-keys" Nov 03 17:17:22.528382 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.528343 5946 cpu_manager.go:414] "containerMap: removing container" podUID="277b8e35-9662-474c-988d-f226d40879e1" containerName="main" Nov 03 17:17:22.528382 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.528351 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277b8e35-9662-474c-988d-f226d40879e1" containerName="main" Nov 03 17:17:22.528822 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.528500 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="277b8e35-9662-474c-988d-f226d40879e1" containerName="main" Nov 03 17:17:22.528822 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.528526 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ffc9e8-bcf1-4f3e-9cc0-bfe2269f345f" containerName="revoke-keys" Nov 03 17:17:22.531771 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.531756 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.603578 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.603542 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:22.628737 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.628701 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-home\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.628929 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.628749 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tmp-dir\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.628929 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.628836 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-model-cache\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.628929 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.628882 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tls-certs\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.629087 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.628974 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4mq\" (UniqueName: \"kubernetes.io/projected/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-kube-api-access-6g4mq\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.629087 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.629037 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-dshm\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.640597 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.640570 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a56144\"/\"kube-root-ca.crt\"" Nov 03 17:17:22.640762 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.640628 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a56144\"/\"default-dockercfg-hbrcb\"" Nov 03 17:17:22.640814 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.640795 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a56144\"/\"openshift-service-ca.crt\"" Nov 03 17:17:22.640905 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.640890 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a56144\"/\"simulator-single-main-a56144-kserve-self-signed-certs\"" Nov 03 17:17:22.642238 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.642198 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd"] Nov 03 17:17:22.729714 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.729676 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4mq\" (UniqueName: \"kubernetes.io/projected/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-kube-api-access-6g4mq\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.729923 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.729719 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-dshm\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.729923 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.729790 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-home\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.729923 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.729836 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tmp-dir\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.729923 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.729863 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-model-cache\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.729923 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.729892 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tls-certs\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.730336 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.730305 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-home\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.730451 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.730319 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tmp-dir\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.730451 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.730366 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-model-cache\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.732545 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.732516 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tls-certs\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.732647 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.732631 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-dshm\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.794308 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.794231 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4mq\" (UniqueName: \"kubernetes.io/projected/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-kube-api-access-6g4mq\") pod \"simulator-single-main-a56144-kserve-b59879df-79txd\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:22.843000 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:22.842960 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:23.265571 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:23.265546 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd"] Nov 03 17:17:23.268251 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:17:23.268222 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ed767d4_dd17_4c5d_8722_06f1fefd0ed0.slice/crio-3851a9b3299671a54229a45ef0ef44563a8b25a74e083b591bdc3e2650d246e1 WatchSource:0}: Error finding container 3851a9b3299671a54229a45ef0ef44563a8b25a74e083b591bdc3e2650d246e1: Status 404 returned error can't find the container with id 3851a9b3299671a54229a45ef0ef44563a8b25a74e083b591bdc3e2650d246e1 Nov 03 17:17:23.292460 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:23.292430 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" event={"ID":"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0","Type":"ContainerStarted","Data":"3851a9b3299671a54229a45ef0ef44563a8b25a74e083b591bdc3e2650d246e1"} Nov 03 17:17:24.298838 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:24.298794 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" event={"ID":"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0","Type":"ContainerStarted","Data":"f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116"} Nov 03 17:17:24.419984 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:24.419925 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" podStartSLOduration=2.419913423 podStartE2EDuration="2.419913423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:24.418239637 +0000 UTC m=+3914.539292458" watchObservedRunningTime="2026-11-03 17:17:24.419913423 +0000 UTC m=+3914.540966245" Nov 03 17:17:24.843670 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:24.843636 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:27.129160 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:27.129129 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:27.142745 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:27.142710 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:17:32.374944 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:32.374916 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:32.843408 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:32.843375 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:32.862411 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:32.862384 5946 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:33.355174 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:33.355142 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:17:37.461240 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:37.461178 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:41.686395 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:41.686358 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:17:42.585313 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:17:42.585278 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:17:42.656850 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:42.656810 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:46.762631 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:46.762584 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:47.792722 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:47.792675 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:53.046695 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:53.046664 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:56.942564 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:56.942528 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56"] Nov 03 17:17:56.946316 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:56.946297 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.008532 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.008502 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:17:57.051903 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.051864 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:17:57.051903 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.051865 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:17:57.052103 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.051911 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:17:57.053584 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.053563 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56"] Nov 03 17:17:57.080982 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.080957 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.081125 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.080992 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhs84\" (UniqueName: \"kubernetes.io/projected/bb783764-b48f-43ed-890f-8c30515abb63-kube-api-access-dhs84\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.081125 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.081054 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.182447 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.182407 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.182643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.182457 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhs84\" (UniqueName: \"kubernetes.io/projected/bb783764-b48f-43ed-890f-8c30515abb63-kube-api-access-dhs84\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.182643 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.182485 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.183184 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.183159 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.184904 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.184880 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.252357 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.252323 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhs84\" (UniqueName: \"kubernetes.io/projected/bb783764-b48f-43ed-890f-8c30515abb63-kube-api-access-dhs84\") pod \"maas-api-revoke-sub-234b8952-59q56\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.259194 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.259170 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:17:57.637480 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:57.637447 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56"] Nov 03 17:17:57.639784 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:17:57.639752 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb783764_b48f_43ed_890f_8c30515abb63.slice/crio-b13173658058f57d0f64951e8355cb40b20b5be963e0ce62c21d3e122384014d WatchSource:0}: Error finding container b13173658058f57d0f64951e8355cb40b20b5be963e0ce62c21d3e122384014d: Status 404 returned error can't find the container with id b13173658058f57d0f64951e8355cb40b20b5be963e0ce62c21d3e122384014d Nov 03 17:17:58.163536 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:58.163498 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:58.464704 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:58.464671 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" event={"ID":"bb783764-b48f-43ed-890f-8c30515abb63","Type":"ContainerStarted","Data":"0c6e6056689c7e7fe7e405419f7e84eed83b665d96d15b466e04e3cdb4e7519b"} Nov 03 17:17:58.464903 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:58.464723 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" event={"ID":"bb783764-b48f-43ed-890f-8c30515abb63","Type":"ContainerStarted","Data":"b13173658058f57d0f64951e8355cb40b20b5be963e0ce62c21d3e122384014d"} Nov 03 17:17:59.471651 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:59.471611 5946 generic.go:356] "Generic (PLEG): container finished" podID="bb783764-b48f-43ed-890f-8c30515abb63" containerID="0c6e6056689c7e7fe7e405419f7e84eed83b665d96d15b466e04e3cdb4e7519b" exitCode=0 Nov 03 17:17:59.472077 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:17:59.471650 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" event={"ID":"bb783764-b48f-43ed-890f-8c30515abb63","Type":"ContainerDied","Data":"0c6e6056689c7e7fe7e405419f7e84eed83b665d96d15b466e04e3cdb4e7519b"} Nov 03 17:18:00.617001 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.616975 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:18:00.719193 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.719158 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-xks-ca\") pod \"bb783764-b48f-43ed-890f-8c30515abb63\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " Nov 03 17:18:00.719389 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.719242 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb783764-b48f-43ed-890f-8c30515abb63-kube-api-access-dhs84\" (UniqueName: \"kubernetes.io/projected/bb783764-b48f-43ed-890f-8c30515abb63-kube-api-access-dhs84\") pod \"bb783764-b48f-43ed-890f-8c30515abb63\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " Nov 03 17:18:00.719389 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.719309 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-openshift-ca\") pod \"bb783764-b48f-43ed-890f-8c30515abb63\" (UID: \"bb783764-b48f-43ed-890f-8c30515abb63\") " Nov 03 17:18:00.719669 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.719647 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-openshift-ca" pod "bb783764-b48f-43ed-890f-8c30515abb63" (UID: "bb783764-b48f-43ed-890f-8c30515abb63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:18:00.721633 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.721606 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-xks-ca" pod "bb783764-b48f-43ed-890f-8c30515abb63" (UID: "bb783764-b48f-43ed-890f-8c30515abb63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:18:00.721756 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.721652 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb783764-b48f-43ed-890f-8c30515abb63-kube-api-access-dhs84" pod "bb783764-b48f-43ed-890f-8c30515abb63" (UID: "bb783764-b48f-43ed-890f-8c30515abb63"). InnerVolumeSpecName "kube-api-access-dhs84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:18:00.821066 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.821028 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:00.821066 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.821059 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb783764-b48f-43ed-890f-8c30515abb63-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:00.821317 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:00.821074 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhs84\" (UniqueName: \"kubernetes.io/projected/bb783764-b48f-43ed-890f-8c30515abb63-kube-api-access-dhs84\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:01.483072 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:01.483035 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" event={"ID":"bb783764-b48f-43ed-890f-8c30515abb63","Type":"ContainerDied","Data":"b13173658058f57d0f64951e8355cb40b20b5be963e0ce62c21d3e122384014d"} Nov 03 17:18:01.483289 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:01.483087 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b13173658058f57d0f64951e8355cb40b20b5be963e0ce62c21d3e122384014d" Nov 03 17:18:01.483289 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:01.483047 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56" Nov 03 17:18:03.440447 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:03.440407 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:08.077111 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.077076 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:08.587077 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.587044 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh"] Nov 03 17:18:08.587590 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.587571 5946 cpu_manager.go:414] "containerMap: removing container" podUID="bb783764-b48f-43ed-890f-8c30515abb63" containerName="revoke-keys" Nov 03 17:18:08.587642 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.587594 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb783764-b48f-43ed-890f-8c30515abb63" containerName="revoke-keys" Nov 03 17:18:08.587698 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.587688 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb783764-b48f-43ed-890f-8c30515abb63" containerName="revoke-keys" Nov 03 17:18:08.593402 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.593370 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.615992 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.615958 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:08.650153 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.650120 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh"] Nov 03 17:18:08.694881 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.694832 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.695062 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.694903 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw74f\" (UniqueName: \"kubernetes.io/projected/ac933367-c2c8-4e3a-b59a-b06792df528f-kube-api-access-kw74f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.695062 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.695023 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.699428 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.699402 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:18:08.699536 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.699452 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:18:08.699536 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.699453 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:18:08.699536 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.699486 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\"" Nov 03 17:18:08.795980 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.795929 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.796193 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.796033 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.796193 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.796082 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw74f\" (UniqueName: \"kubernetes.io/projected/ac933367-c2c8-4e3a-b59a-b06792df528f-kube-api-access-kw74f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.796709 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.796687 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.798631 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.798608 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.856932 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.856860 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw74f\" (UniqueName: \"kubernetes.io/projected/ac933367-c2c8-4e3a-b59a-b06792df528f-kube-api-access-kw74f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:08.915022 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:08.914974 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:09.295079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:09.295049 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh"] Nov 03 17:18:09.297725 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:18:09.297690 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac933367_c2c8_4e3a_b59a_b06792df528f.slice/crio-f2c6006bf390c4c2c8472707b780327e8ca724458229c7e14557214572763bd8 WatchSource:0}: Error finding container f2c6006bf390c4c2c8472707b780327e8ca724458229c7e14557214572763bd8: Status 404 returned error can't find the container with id f2c6006bf390c4c2c8472707b780327e8ca724458229c7e14557214572763bd8 Nov 03 17:18:09.523257 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:09.523221 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" event={"ID":"ac933367-c2c8-4e3a-b59a-b06792df528f","Type":"ContainerStarted","Data":"f2c6006bf390c4c2c8472707b780327e8ca724458229c7e14557214572763bd8"} Nov 03 17:18:09.887460 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:09.887367 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:18:10.530444 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:10.530408 5946 generic.go:356] "Generic (PLEG): container finished" podID="ac933367-c2c8-4e3a-b59a-b06792df528f" containerID="ac2ec104d6098bbe86c7ef71babefadc6a8fefa134b8703023276ac7521144a7" exitCode=0 Nov 03 17:18:10.530834 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:10.530489 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" event={"ID":"ac933367-c2c8-4e3a-b59a-b06792df528f","Type":"ContainerDied","Data":"ac2ec104d6098bbe86c7ef71babefadc6a8fefa134b8703023276ac7521144a7"} Nov 03 17:18:11.671860 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.671838 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:11.722428 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.722391 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-openshift-ca\") pod \"ac933367-c2c8-4e3a-b59a-b06792df528f\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " Nov 03 17:18:11.722608 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.722469 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac933367-c2c8-4e3a-b59a-b06792df528f-kube-api-access-kw74f\" (UniqueName: \"kubernetes.io/projected/ac933367-c2c8-4e3a-b59a-b06792df528f-kube-api-access-kw74f\") pod \"ac933367-c2c8-4e3a-b59a-b06792df528f\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " Nov 03 17:18:11.722608 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.722503 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-xks-ca\") pod \"ac933367-c2c8-4e3a-b59a-b06792df528f\" (UID: \"ac933367-c2c8-4e3a-b59a-b06792df528f\") " Nov 03 17:18:11.722770 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.722740 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-openshift-ca" pod "ac933367-c2c8-4e3a-b59a-b06792df528f" (UID: "ac933367-c2c8-4e3a-b59a-b06792df528f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:18:11.724624 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.724602 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-xks-ca" pod "ac933367-c2c8-4e3a-b59a-b06792df528f" (UID: "ac933367-c2c8-4e3a-b59a-b06792df528f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:18:11.724745 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.724692 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac933367-c2c8-4e3a-b59a-b06792df528f-kube-api-access-kw74f" pod "ac933367-c2c8-4e3a-b59a-b06792df528f" (UID: "ac933367-c2c8-4e3a-b59a-b06792df528f"). InnerVolumeSpecName "kube-api-access-kw74f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:18:11.824162 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.824067 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:11.824162 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.824099 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw74f\" (UniqueName: \"kubernetes.io/projected/ac933367-c2c8-4e3a-b59a-b06792df528f-kube-api-access-kw74f\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:11.824162 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:11.824113 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac933367-c2c8-4e3a-b59a-b06792df528f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:12.541641 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:12.541608 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" Nov 03 17:18:12.541804 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:12.541646 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh" event={"ID":"ac933367-c2c8-4e3a-b59a-b06792df528f","Type":"ContainerDied","Data":"f2c6006bf390c4c2c8472707b780327e8ca724458229c7e14557214572763bd8"} Nov 03 17:18:12.541804 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:12.541696 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2c6006bf390c4c2c8472707b780327e8ca724458229c7e14557214572763bd8" Nov 03 17:18:12.585021 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:18:12.584983 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:18:13.007380 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:13.007347 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd"] Nov 03 17:18:13.132304 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:13.132263 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:14.492734 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:14.492700 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:19.670468 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:19.670435 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:20.440551 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:20.440515 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:20.640871 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:20.640840 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:21.066505 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:21.066464 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb"] Nov 03 17:18:21.117285 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:21.117252 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68d2fd7-4m2nb"] Nov 03 17:18:22.002425 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:22.002394 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3d7ee5-43d0-450a-89a2-acbaf4accfd8" path="/var/lib/kubelet/pods/ca3d7ee5-43d0-450a-89a2-acbaf4accfd8/volumes" Nov 03 17:18:24.204830 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:24.204800 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:25.800442 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:25.800398 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:26.092126 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:26.092039 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:26.446080 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:26.445992 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh"] Nov 03 17:18:26.496746 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:26.496709 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a56144-v9klh"] Nov 03 17:18:26.852307 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:26.852272 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:28.002980 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.002946 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac933367-c2c8-4e3a-b59a-b06792df528f" path="/var/lib/kubelet/pods/ac933367-c2c8-4e3a-b59a-b06792df528f/volumes" Nov 03 17:18:28.027596 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.027551 5946 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" podUID="4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" containerName="main" containerID="cri-o://f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116" gracePeriod=45 Nov 03 17:18:28.280481 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.280455 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:18:28.377186 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377146 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-model-cache\") pod \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " Nov 03 17:18:28.377413 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377235 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tls-certs\") pod \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " Nov 03 17:18:28.377413 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377348 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-kube-api-access-6g4mq\" (UniqueName: \"kubernetes.io/projected/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-kube-api-access-6g4mq\") pod \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " Nov 03 17:18:28.377413 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377389 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tmp-dir\") pod \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " Nov 03 17:18:28.377600 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377447 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-dshm\") pod \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " Nov 03 17:18:28.377600 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377482 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-model-cache" pod "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" (UID: "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:18:28.377600 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377505 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-home\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-home\") pod \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\" (UID: \"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0\") " Nov 03 17:18:28.377819 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377718 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tmp-dir" pod "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" (UID: "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:18:28.377943 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377918 5946 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tmp-dir\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:28.378095 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377946 5946 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-model-cache\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:28.378095 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.377941 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-home" pod "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" (UID: "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:18:28.379820 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.379800 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-dshm" pod "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" (UID: "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:18:28.380074 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.380054 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tls-certs" pod "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" (UID: "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:18:28.380158 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.380095 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-kube-api-access-6g4mq" pod "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" (UID: "4ed767d4-dd17-4c5d-8722-06f1fefd0ed0"). InnerVolumeSpecName "kube-api-access-6g4mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:18:28.479173 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.479136 5946 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-tls-certs\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:28.479173 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.479171 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g4mq\" (UniqueName: \"kubernetes.io/projected/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-kube-api-access-6g4mq\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:28.479403 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.479183 5946 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-dshm\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:28.479403 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.479190 5946 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0-home\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:28.571454 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.571346 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:18:28.618884 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.618850 5946 generic.go:356] "Generic (PLEG): container finished" podID="4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" containerID="f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116" exitCode=0 Nov 03 17:18:28.619099 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.618896 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" Nov 03 17:18:28.619099 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.618939 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" event={"ID":"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0","Type":"ContainerDied","Data":"f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116"} Nov 03 17:18:28.619099 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.618981 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd" event={"ID":"4ed767d4-dd17-4c5d-8722-06f1fefd0ed0","Type":"ContainerDied","Data":"3851a9b3299671a54229a45ef0ef44563a8b25a74e083b591bdc3e2650d246e1"} Nov 03 17:18:28.619099 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.619006 5946 scope.go:122] "RemoveContainer" containerID="f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116" Nov 03 17:18:28.629220 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.629188 5946 scope.go:122] "RemoveContainer" containerID="f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116" Nov 03 17:18:28.629569 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:18:28.629550 5946 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116\": container with ID starting with f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116 not found: ID does not exist" containerID="f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116" Nov 03 17:18:28.629625 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.629583 5946 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116"} err="failed to get container status \"f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116\": rpc error: code = NotFound desc = could not find container \"f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116\": container with ID starting with f02ea77a0744a21db4d718fed2611f2cfbc64f5d8f3c25631a1e4c05d36f4116 not found: ID does not exist" Nov 03 17:18:28.790225 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.790171 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd"] Nov 03 17:18:28.792294 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:28.792270 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a56144/simulator-single-main-a56144-kserve-b59879df-79txd"] Nov 03 17:18:30.002925 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:30.002893 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" path="/var/lib/kubelet/pods/4ed767d4-dd17-4c5d-8722-06f1fefd0ed0/volumes" Nov 03 17:18:33.584870 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:33.584833 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:33.776358 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:33.776322 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:34.002735 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:34.002695 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:40.005237 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:40.005143 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:42.585494 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:18:42.585450 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:18:44.351116 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:44.351083 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:18:46.838233 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:46.829306 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:18:46.838742 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:46.838661 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:18:47.102846 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:47.102766 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:47.624428 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:47.624384 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-kw9pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:18:47.624883 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:47.624840 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-kw9pt" podUID="f2070296-e232-4796-a766-50240d2a2286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:18:50.204226 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:50.204183 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:51.622131 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:18:51.622099 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:08.469843 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:19:08.469806 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:12.585360 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:19:12.585321 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:19:23.479911 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:19:23.479874 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:27.882435 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:19:27.882396 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:19:31.346862 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:19:31.346822 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:19:31.347599 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:19:31.347548 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:19:42.585651 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:19:42.585613 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:19:47.445714 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:19:47.445662 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:58.679125 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:19:58.679089 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:20:01.371891 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:01.371332 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:20:01.371891 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:01.371445 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:20:01.832233 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:01.828326 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:20:01.832233 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:01.828396 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:20:10.257998 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:10.257956 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:12.585700 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:20:12.585665 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:20:13.070806 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:13.070774 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:13.338849 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:13.338766 5946 scope.go:122] "RemoveContainer" containerID="14d21304faeb49e76d3948a6f188b7d6c7388eb9f89e39df868766e77d96436f" Nov 03 17:20:13.347641 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:13.347615 5946 scope.go:122] "RemoveContainer" containerID="a4ea325f76c21524878263c6015693a849f65c3d7718b949bf8bb686dff45dfd" Nov 03 17:20:13.789731 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:13.789695 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:20:16.380383 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:16.380332 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-w8lvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:20:16.381061 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:16.381022 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-w8lvg" podUID="72b9663c-c665-4a5a-a2c7-c564bd2696b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:20:16.831384 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:16.831343 5946 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcq7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:20:16.831740 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:16.831691 5946 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcq7f" podUID="1d902919-9525-449c-9b32-c58e0fde1618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:20:21.731301 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:21.731232 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:25.544684 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:25.544649 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:20:33.398981 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:33.398933 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:36.879818 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:36.879779 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:20:42.584518 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:20:42.584468 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:20:47.784706 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:47.784668 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:48.091364 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:20:48.091267 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:21:08.754408 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:08.754372 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:21:12.584584 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:21:12.584545 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:21:15.649184 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:15.649142 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:17.458086 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:17.458009 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:19.762637 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:19.762565 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:20.000424 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:20.000379 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:22.985735 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:22.985695 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:23.180111 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:23.180064 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:25.848693 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:25.848663 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:26.050359 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:26.050321 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:26.254619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:26.254586 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:31.347614 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:31.347576 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:33.981876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:33.981841 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:21:36.652409 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:36.652375 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:37.302990 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:37.302952 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:41.832593 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:41.832557 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:42.584843 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:21:42.584808 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:21:47.592400 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:47.592318 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:48.113139 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:48.113101 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:52.974725 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:52.974690 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:56.084287 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.084252 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq"] Nov 03 17:21:56.084726 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.084653 5946 cpu_manager.go:414] "containerMap: removing container" podUID="4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" containerName="main" Nov 03 17:21:56.084726 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.084664 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" containerName="main" Nov 03 17:21:56.084726 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.084688 5946 cpu_manager.go:414] "containerMap: removing container" podUID="ac933367-c2c8-4e3a-b59a-b06792df528f" containerName="revoke-keys" Nov 03 17:21:56.084726 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.084693 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac933367-c2c8-4e3a-b59a-b06792df528f" containerName="revoke-keys" Nov 03 17:21:56.084896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.084742 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac933367-c2c8-4e3a-b59a-b06792df528f" containerName="revoke-keys" Nov 03 17:21:56.084896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.084756 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed767d4-dd17-4c5d-8722-06f1fefd0ed0" containerName="main" Nov 03 17:21:56.087796 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.087781 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.147818 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.147778 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq"] Nov 03 17:21:56.195799 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.195762 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8g7h\" (UniqueName: \"kubernetes.io/projected/202dcee3-6a94-434d-a85b-caf47a995ffd-kube-api-access-z8g7h\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.195977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.195877 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.195977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.195950 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.196533 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.196516 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:21:56.196590 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.196523 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:21:56.196590 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.196542 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:21:56.196590 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.196584 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6jprr\"" Nov 03 17:21:56.296901 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.296865 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8g7h\" (UniqueName: \"kubernetes.io/projected/202dcee3-6a94-434d-a85b-caf47a995ffd-kube-api-access-z8g7h\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.297079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.296947 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.297079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.296994 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.297612 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.297593 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.299525 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.299501 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.357957 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.357877 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8g7h\" (UniqueName: \"kubernetes.io/projected/202dcee3-6a94-434d-a85b-caf47a995ffd-kube-api-access-z8g7h\") pod \"maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.400820 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.400786 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:56.775508 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:56.775485 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq"] Nov 03 17:21:56.778085 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:21:56.778060 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202dcee3_6a94_434d_a85b_caf47a995ffd.slice/crio-7dfc215b6a012253c4b5b6f5892984ee432e4f641bb6bd2e0bac791a796cf641 WatchSource:0}: Error finding container 7dfc215b6a012253c4b5b6f5892984ee432e4f641bb6bd2e0bac791a796cf641: Status 404 returned error can't find the container with id 7dfc215b6a012253c4b5b6f5892984ee432e4f641bb6bd2e0bac791a796cf641 Nov 03 17:21:57.759987 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:57.759953 5946 generic.go:356] "Generic (PLEG): container finished" podID="202dcee3-6a94-434d-a85b-caf47a995ffd" containerID="94359950a7bcbafce0f44fa9f814cc03291abac9352a4678528d26525112b6cf" exitCode=0 Nov 03 17:21:57.760431 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:57.759993 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" event={"ID":"202dcee3-6a94-434d-a85b-caf47a995ffd","Type":"ContainerDied","Data":"94359950a7bcbafce0f44fa9f814cc03291abac9352a4678528d26525112b6cf"} Nov 03 17:21:57.760431 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:57.760043 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" event={"ID":"202dcee3-6a94-434d-a85b-caf47a995ffd","Type":"ContainerStarted","Data":"7dfc215b6a012253c4b5b6f5892984ee432e4f641bb6bd2e0bac791a796cf641"} Nov 03 17:21:58.151277 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.151243 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:58.894092 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.894070 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:58.920129 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.920105 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202dcee3-6a94-434d-a85b-caf47a995ffd-kube-api-access-z8g7h\" (UniqueName: \"kubernetes.io/projected/202dcee3-6a94-434d-a85b-caf47a995ffd-kube-api-access-z8g7h\") pod \"202dcee3-6a94-434d-a85b-caf47a995ffd\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " Nov 03 17:21:58.920299 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.920161 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-openshift-ca\") pod \"202dcee3-6a94-434d-a85b-caf47a995ffd\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " Nov 03 17:21:58.920299 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.920279 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-xks-ca\") pod \"202dcee3-6a94-434d-a85b-caf47a995ffd\" (UID: \"202dcee3-6a94-434d-a85b-caf47a995ffd\") " Nov 03 17:21:58.920570 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.920545 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-openshift-ca" pod "202dcee3-6a94-434d-a85b-caf47a995ffd" (UID: "202dcee3-6a94-434d-a85b-caf47a995ffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:21:58.922480 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.922452 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-xks-ca" pod "202dcee3-6a94-434d-a85b-caf47a995ffd" (UID: "202dcee3-6a94-434d-a85b-caf47a995ffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:21:58.922570 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:58.922456 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202dcee3-6a94-434d-a85b-caf47a995ffd-kube-api-access-z8g7h" pod "202dcee3-6a94-434d-a85b-caf47a995ffd" (UID: "202dcee3-6a94-434d-a85b-caf47a995ffd"). InnerVolumeSpecName "kube-api-access-z8g7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:59.022058 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:59.021967 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:21:59.022058 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:59.022002 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8g7h\" (UniqueName: \"kubernetes.io/projected/202dcee3-6a94-434d-a85b-caf47a995ffd-kube-api-access-z8g7h\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:21:59.022058 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:59.022016 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202dcee3-6a94-434d-a85b-caf47a995ffd-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:21:59.771255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:59.771221 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" Nov 03 17:21:59.771255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:59.771234 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq" event={"ID":"202dcee3-6a94-434d-a85b-caf47a995ffd","Type":"ContainerDied","Data":"7dfc215b6a012253c4b5b6f5892984ee432e4f641bb6bd2e0bac791a796cf641"} Nov 03 17:21:59.771485 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:21:59.771284 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dfc215b6a012253c4b5b6f5892984ee432e4f641bb6bd2e0bac791a796cf641" Nov 03 17:22:03.347335 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:03.347281 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:07.902415 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:07.902384 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:08.138462 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:08.138422 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:10.657149 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:10.657123 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:22:10.676463 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:10.676431 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:22:11.870724 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:11.870693 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:12.137877 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:12.137788 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:22:12.155584 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:12.155546 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:12.399328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:12.399237 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq"] Nov 03 17:22:12.403897 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:12.403872 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-583a0c6c-mvmnq"] Nov 03 17:22:12.584899 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:22:12.584859 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:22:14.003243 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:14.003174 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202dcee3-6a94-434d-a85b-caf47a995ffd" path="/var/lib/kubelet/pods/202dcee3-6a94-434d-a85b-caf47a995ffd/volumes" Nov 03 17:22:17.180684 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:17.180647 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:28.439030 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:28.438996 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:29.083328 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:29.083284 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:29.301291 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:29.301254 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:33.700766 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:33.700737 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:33.891693 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:33.891663 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:36.945918 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:36.945880 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:37.139348 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:37.139317 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:40.393440 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:40.393388 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:40.580694 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:40.580638 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:40.800306 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:40.800274 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:42.585264 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:22:42.585227 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:22:45.667848 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:45.667809 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:48.370175 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:48.370129 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:22:48.508747 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:48.508708 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:50.950087 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:50.950052 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:51.260947 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:51.260914 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:56.070813 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:56.070774 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:58.125025 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:22:58.124965 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:01.062241 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:01.062189 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56"] Nov 03 17:23:01.112850 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:01.112813 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234b8952-59q56"] Nov 03 17:23:01.558869 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:01.558836 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:02.002497 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:02.002461 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb783764-b48f-43ed-890f-8c30515abb63" path="/var/lib/kubelet/pods/bb783764-b48f-43ed-890f-8c30515abb63/volumes" Nov 03 17:23:06.752705 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:06.752652 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:11.937451 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:11.937400 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:12.584321 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:23:12.584284 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:23:17.041341 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:17.041299 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:22.274135 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:22.274095 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:27.344143 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:27.344105 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:32.525370 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:32.525287 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:37.639014 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:37.638971 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:42.585358 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:23:42.585317 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:23:42.842397 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:42.842312 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:47.959682 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:47.959643 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:48.797045 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:48.797010 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:53.154607 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:53.154567 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:58.339938 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:23:58.339891 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:03.648083 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:03.648019 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:03.935123 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:03.935036 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:04.132410 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:04.132373 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:08.754151 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:08.754119 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:12.584907 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:24:12.584865 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:24:13.402902 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:13.402870 5946 scope.go:122] "RemoveContainer" containerID="0c6e6056689c7e7fe7e405419f7e84eed83b665d96d15b466e04e3cdb4e7519b" Nov 03 17:24:13.411811 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:13.411787 5946 scope.go:122] "RemoveContainer" containerID="ac2ec104d6098bbe86c7ef71babefadc6a8fefa134b8703023276ac7521144a7" Nov 03 17:24:13.930888 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:13.930857 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:19.155279 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:19.155247 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:21.451276 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.451238 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988"] Nov 03 17:24:21.451685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.451628 5946 cpu_manager.go:414] "containerMap: removing container" podUID="202dcee3-6a94-434d-a85b-caf47a995ffd" containerName="revoke-keys" Nov 03 17:24:21.451685 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.451639 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202dcee3-6a94-434d-a85b-caf47a995ffd" containerName="revoke-keys" Nov 03 17:24:21.451760 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.451725 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="202dcee3-6a94-434d-a85b-caf47a995ffd" containerName="revoke-keys" Nov 03 17:24:21.456396 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.456375 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.516132 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.516103 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988"] Nov 03 17:24:21.565363 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.565324 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:24:21.565363 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.565337 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:24:21.565363 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.565347 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:24:21.565623 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.565422 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6jprr\"" Nov 03 17:24:21.573828 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.573787 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.573967 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.573871 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.573967 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.573956 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42wtz\" (UniqueName: \"kubernetes.io/projected/079916e2-cbac-460a-a5ba-b8ee2d50621f-kube-api-access-42wtz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.675270 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.675201 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.675460 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.675309 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42wtz\" (UniqueName: \"kubernetes.io/projected/079916e2-cbac-460a-a5ba-b8ee2d50621f-kube-api-access-42wtz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.675511 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.675463 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.675953 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.675937 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.677794 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.677777 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.736315 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.736286 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42wtz\" (UniqueName: \"kubernetes.io/projected/079916e2-cbac-460a-a5ba-b8ee2d50621f-kube-api-access-42wtz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:21.769773 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:21.769743 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:22.149527 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:22.149500 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988"] Nov 03 17:24:22.151436 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:24:22.151408 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079916e2_cbac_460a_a5ba_b8ee2d50621f.slice/crio-eeb74181861e8bae1b1e300b3b113f38e4a0b5c283d271c44a67c25d04b9a618 WatchSource:0}: Error finding container eeb74181861e8bae1b1e300b3b113f38e4a0b5c283d271c44a67c25d04b9a618: Status 404 returned error can't find the container with id eeb74181861e8bae1b1e300b3b113f38e4a0b5c283d271c44a67c25d04b9a618 Nov 03 17:24:22.463148 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:22.463115 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" event={"ID":"079916e2-cbac-460a-a5ba-b8ee2d50621f","Type":"ContainerStarted","Data":"490b53aecf1cfae71f7b82cc9eaaf80fe57192f9aeb83f8d7997463396333d23"} Nov 03 17:24:22.463528 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:22.463169 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" event={"ID":"079916e2-cbac-460a-a5ba-b8ee2d50621f","Type":"ContainerStarted","Data":"eeb74181861e8bae1b1e300b3b113f38e4a0b5c283d271c44a67c25d04b9a618"} Nov 03 17:24:22.583530 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:22.583479 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" podStartSLOduration=1.58346662 podStartE2EDuration="1.58346662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:24:22.58235935 +0000 UTC m=+4332.703412173" watchObservedRunningTime="2026-11-03 17:24:22.58346662 +0000 UTC m=+4332.704519443" Nov 03 17:24:23.470021 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:23.469988 5946 generic.go:356] "Generic (PLEG): container finished" podID="079916e2-cbac-460a-a5ba-b8ee2d50621f" containerID="490b53aecf1cfae71f7b82cc9eaaf80fe57192f9aeb83f8d7997463396333d23" exitCode=0 Nov 03 17:24:23.470458 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:23.470076 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" event={"ID":"079916e2-cbac-460a-a5ba-b8ee2d50621f","Type":"ContainerDied","Data":"490b53aecf1cfae71f7b82cc9eaaf80fe57192f9aeb83f8d7997463396333d23"} Nov 03 17:24:24.606876 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.606837 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:24.610909 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.610879 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:24.703262 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.703179 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-xks-ca\") pod \"079916e2-cbac-460a-a5ba-b8ee2d50621f\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " Nov 03 17:24:24.703439 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.703272 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-openshift-ca\") pod \"079916e2-cbac-460a-a5ba-b8ee2d50621f\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " Nov 03 17:24:24.703439 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.703332 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079916e2-cbac-460a-a5ba-b8ee2d50621f-kube-api-access-42wtz\" (UniqueName: \"kubernetes.io/projected/079916e2-cbac-460a-a5ba-b8ee2d50621f-kube-api-access-42wtz\") pod \"079916e2-cbac-460a-a5ba-b8ee2d50621f\" (UID: \"079916e2-cbac-460a-a5ba-b8ee2d50621f\") " Nov 03 17:24:24.703633 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.703610 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-openshift-ca" pod "079916e2-cbac-460a-a5ba-b8ee2d50621f" (UID: "079916e2-cbac-460a-a5ba-b8ee2d50621f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:24.705539 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.705511 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-xks-ca" pod "079916e2-cbac-460a-a5ba-b8ee2d50621f" (UID: "079916e2-cbac-460a-a5ba-b8ee2d50621f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:24.705665 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.705588 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079916e2-cbac-460a-a5ba-b8ee2d50621f-kube-api-access-42wtz" pod "079916e2-cbac-460a-a5ba-b8ee2d50621f" (UID: "079916e2-cbac-460a-a5ba-b8ee2d50621f"). InnerVolumeSpecName "kube-api-access-42wtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:24.804868 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.804833 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:24:24.804868 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.804864 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079916e2-cbac-460a-a5ba-b8ee2d50621f-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:24:24.804868 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:24.804875 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42wtz\" (UniqueName: \"kubernetes.io/projected/079916e2-cbac-460a-a5ba-b8ee2d50621f-kube-api-access-42wtz\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:24:25.480957 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:25.480921 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" event={"ID":"079916e2-cbac-460a-a5ba-b8ee2d50621f","Type":"ContainerDied","Data":"eeb74181861e8bae1b1e300b3b113f38e4a0b5c283d271c44a67c25d04b9a618"} Nov 03 17:24:25.481131 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:25.480973 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb74181861e8bae1b1e300b3b113f38e4a0b5c283d271c44a67c25d04b9a618" Nov 03 17:24:25.481131 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:25.480937 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988" Nov 03 17:24:29.944495 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:29.944462 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:33.374096 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:33.374037 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:33.569654 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:33.569616 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:39.530418 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:39.530368 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:41.137920 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:41.137886 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:41.418900 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:41.418817 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:41.770303 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:41.770274 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988"] Nov 03 17:24:41.821078 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:41.821049 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-52cc05-6r988"] Nov 03 17:24:42.002228 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:42.002171 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079916e2-cbac-460a-a5ba-b8ee2d50621f" path="/var/lib/kubelet/pods/079916e2-cbac-460a-a5ba-b8ee2d50621f/volumes" Nov 03 17:24:42.132864 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:42.132781 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:42.585006 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:24:42.584964 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:24:45.575016 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:45.574927 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:24:48.872964 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:48.872925 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:49.064256 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:49.064221 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:49.259071 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:49.259036 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:57.261553 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:24:57.261518 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:25:08.277928 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:08.271892 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:25:12.585074 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:25:12.585026 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:25:16.465106 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.465056 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx"] Nov 03 17:25:16.465627 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.465507 5946 cpu_manager.go:414] "containerMap: removing container" podUID="079916e2-cbac-460a-a5ba-b8ee2d50621f" containerName="revoke-keys" Nov 03 17:25:16.465627 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.465520 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079916e2-cbac-460a-a5ba-b8ee2d50621f" containerName="revoke-keys" Nov 03 17:25:16.465627 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.465613 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="079916e2-cbac-460a-a5ba-b8ee2d50621f" containerName="revoke-keys" Nov 03 17:25:16.468863 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.468846 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.542807 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.542774 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mz4w\" (UniqueName: \"kubernetes.io/projected/831a0d73-e900-424e-a8b3-cef70a821cf9-kube-api-access-7mz4w\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.543020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.542832 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.543020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.542922 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.574573 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.574543 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:25:16.574573 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.574555 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:25:16.574823 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.574585 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:25:16.574823 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.574667 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6jprr\"" Nov 03 17:25:16.576468 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.576446 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx"] Nov 03 17:25:16.643798 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.643759 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.644009 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.643835 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.644009 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.643915 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mz4w\" (UniqueName: \"kubernetes.io/projected/831a0d73-e900-424e-a8b3-cef70a821cf9-kube-api-access-7mz4w\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.644608 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.644511 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.646301 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.646275 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.705018 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.704990 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mz4w\" (UniqueName: \"kubernetes.io/projected/831a0d73-e900-424e-a8b3-cef70a821cf9-kube-api-access-7mz4w\") pod \"maas-api-revoke-sub-f03629db-d9wtx\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:16.782372 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:16.782337 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:17.161907 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:17.161884 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx"] Nov 03 17:25:17.164155 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:25:17.164127 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831a0d73_e900_424e_a8b3_cef70a821cf9.slice/crio-ea3541017010ee9f2dc44c75cc56c545e7e81bc7e4f77331bc7e9468367fd3fa WatchSource:0}: Error finding container ea3541017010ee9f2dc44c75cc56c545e7e81bc7e4f77331bc7e9468367fd3fa: Status 404 returned error can't find the container with id ea3541017010ee9f2dc44c75cc56c545e7e81bc7e4f77331bc7e9468367fd3fa Nov 03 17:25:17.793776 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:17.793738 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" event={"ID":"831a0d73-e900-424e-a8b3-cef70a821cf9","Type":"ContainerStarted","Data":"b4ad7f41479b32b9300b40f9505da59be4a5af37de4e363cda6f7d1e3d1233db"} Nov 03 17:25:17.794189 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:17.793796 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" event={"ID":"831a0d73-e900-424e-a8b3-cef70a821cf9","Type":"ContainerStarted","Data":"ea3541017010ee9f2dc44c75cc56c545e7e81bc7e4f77331bc7e9468367fd3fa"} Nov 03 17:25:17.915325 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:17.915272 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" podStartSLOduration=1.915258665 podStartE2EDuration="1.915258665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:25:17.913945088 +0000 UTC m=+4388.034997911" watchObservedRunningTime="2026-11-03 17:25:17.915258665 +0000 UTC m=+4388.036311487" Nov 03 17:25:18.799642 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:18.799606 5946 generic.go:356] "Generic (PLEG): container finished" podID="831a0d73-e900-424e-a8b3-cef70a821cf9" containerID="b4ad7f41479b32b9300b40f9505da59be4a5af37de4e363cda6f7d1e3d1233db" exitCode=0 Nov 03 17:25:18.800048 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:18.799664 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" event={"ID":"831a0d73-e900-424e-a8b3-cef70a821cf9","Type":"ContainerDied","Data":"b4ad7f41479b32b9300b40f9505da59be4a5af37de4e363cda6f7d1e3d1233db"} Nov 03 17:25:19.939672 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.939642 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:19.976085 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.976037 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-openshift-ca\") pod \"831a0d73-e900-424e-a8b3-cef70a821cf9\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " Nov 03 17:25:19.976255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.976136 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-xks-ca\") pod \"831a0d73-e900-424e-a8b3-cef70a821cf9\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " Nov 03 17:25:19.976255 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.976197 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831a0d73-e900-424e-a8b3-cef70a821cf9-kube-api-access-7mz4w\" (UniqueName: \"kubernetes.io/projected/831a0d73-e900-424e-a8b3-cef70a821cf9-kube-api-access-7mz4w\") pod \"831a0d73-e900-424e-a8b3-cef70a821cf9\" (UID: \"831a0d73-e900-424e-a8b3-cef70a821cf9\") " Nov 03 17:25:19.976450 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.976428 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-openshift-ca" pod "831a0d73-e900-424e-a8b3-cef70a821cf9" (UID: "831a0d73-e900-424e-a8b3-cef70a821cf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:19.976630 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.976606 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:19.978386 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.978359 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-xks-ca" pod "831a0d73-e900-424e-a8b3-cef70a821cf9" (UID: "831a0d73-e900-424e-a8b3-cef70a821cf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:19.978501 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:19.978482 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831a0d73-e900-424e-a8b3-cef70a821cf9-kube-api-access-7mz4w" pod "831a0d73-e900-424e-a8b3-cef70a821cf9" (UID: "831a0d73-e900-424e-a8b3-cef70a821cf9"). InnerVolumeSpecName "kube-api-access-7mz4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:20.078825 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:20.078554 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831a0d73-e900-424e-a8b3-cef70a821cf9-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:20.078825 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:20.078589 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mz4w\" (UniqueName: \"kubernetes.io/projected/831a0d73-e900-424e-a8b3-cef70a821cf9-kube-api-access-7mz4w\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:20.722220 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:20.722175 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:20.811058 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:20.811023 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" event={"ID":"831a0d73-e900-424e-a8b3-cef70a821cf9","Type":"ContainerDied","Data":"ea3541017010ee9f2dc44c75cc56c545e7e81bc7e4f77331bc7e9468367fd3fa"} Nov 03 17:25:20.811294 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:20.811063 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx" Nov 03 17:25:20.811294 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:20.811070 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3541017010ee9f2dc44c75cc56c545e7e81bc7e4f77331bc7e9468367fd3fa" Nov 03 17:25:21.254240 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:21.254177 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:21.449750 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:21.449714 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:29.283451 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:29.283311 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:25:37.380805 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.380767 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs"] Nov 03 17:25:37.381401 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.381380 5946 cpu_manager.go:414] "containerMap: removing container" podUID="831a0d73-e900-424e-a8b3-cef70a821cf9" containerName="revoke-keys" Nov 03 17:25:37.381491 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.381403 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831a0d73-e900-424e-a8b3-cef70a821cf9" containerName="revoke-keys" Nov 03 17:25:37.381579 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.381566 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="831a0d73-e900-424e-a8b3-cef70a821cf9" containerName="revoke-keys" Nov 03 17:25:37.384960 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.384939 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.443057 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.443020 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs"] Nov 03 17:25:37.469079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.469046 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.469273 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.469091 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.469273 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.469112 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkb44\" (UniqueName: \"kubernetes.io/projected/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-kube-api-access-kkb44\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.492218 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.492184 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:25:37.492366 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.492201 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:25:37.492366 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.492266 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:25:37.492366 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.492311 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6jprr\"" Nov 03 17:25:37.569969 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.569941 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.570128 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.569996 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.570128 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.570020 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkb44\" (UniqueName: \"kubernetes.io/projected/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-kube-api-access-kkb44\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.570601 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.570582 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.572436 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.572413 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.632666 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.632591 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkb44\" (UniqueName: \"kubernetes.io/projected/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-kube-api-access-kkb44\") pod \"maas-api-revoke-sub-0165d7d2-f2xrs\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:37.696789 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:37.696751 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:38.071979 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:38.071948 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs"] Nov 03 17:25:38.074407 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:25:38.074375 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d39c8bb_9e3d_49e6_acc7_43a20ce2502d.slice/crio-70f81662bf2e92f4fc1ddca6dd01011637f92f6cd6f613cf176480b12a591480 WatchSource:0}: Error finding container 70f81662bf2e92f4fc1ddca6dd01011637f92f6cd6f613cf176480b12a591480: Status 404 returned error can't find the container with id 70f81662bf2e92f4fc1ddca6dd01011637f92f6cd6f613cf176480b12a591480 Nov 03 17:25:38.903590 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:38.903552 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" event={"ID":"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d","Type":"ContainerStarted","Data":"e7ad921482365088367de01772fb9d1bdf27b13b519cb5e1dce0618362fcb89e"} Nov 03 17:25:38.904040 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:38.903607 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" event={"ID":"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d","Type":"ContainerStarted","Data":"70f81662bf2e92f4fc1ddca6dd01011637f92f6cd6f613cf176480b12a591480"} Nov 03 17:25:39.034011 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:39.033957 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" podStartSLOduration=2.033944988 podStartE2EDuration="2.033944988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:25:39.03259181 +0000 UTC m=+4409.153644634" watchObservedRunningTime="2026-11-03 17:25:39.033944988 +0000 UTC m=+4409.154997804" Nov 03 17:25:39.910114 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:39.910072 5946 generic.go:356] "Generic (PLEG): container finished" podID="1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" containerID="e7ad921482365088367de01772fb9d1bdf27b13b519cb5e1dce0618362fcb89e" exitCode=0 Nov 03 17:25:39.910602 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:39.910162 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" event={"ID":"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d","Type":"ContainerDied","Data":"e7ad921482365088367de01772fb9d1bdf27b13b519cb5e1dce0618362fcb89e"} Nov 03 17:25:41.048483 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.048458 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:41.206420 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.206326 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-kube-api-access-kkb44\" (UniqueName: \"kubernetes.io/projected/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-kube-api-access-kkb44\") pod \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " Nov 03 17:25:41.206611 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.206470 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-xks-ca\") pod \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " Nov 03 17:25:41.206611 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.206544 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-openshift-ca\") pod \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\" (UID: \"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d\") " Nov 03 17:25:41.206898 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.206874 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-openshift-ca" pod "1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" (UID: "1d39c8bb-9e3d-49e6-acc7-43a20ce2502d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:41.208627 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.208604 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-kube-api-access-kkb44" pod "1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" (UID: "1d39c8bb-9e3d-49e6-acc7-43a20ce2502d"). InnerVolumeSpecName "kube-api-access-kkb44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:41.208710 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.208649 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-xks-ca" pod "1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" (UID: "1d39c8bb-9e3d-49e6-acc7-43a20ce2502d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:41.308265 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.308222 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkb44\" (UniqueName: \"kubernetes.io/projected/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-kube-api-access-kkb44\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:41.308265 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.308255 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:41.308265 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.308268 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:41.923903 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.923860 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" event={"ID":"1d39c8bb-9e3d-49e6-acc7-43a20ce2502d","Type":"ContainerDied","Data":"70f81662bf2e92f4fc1ddca6dd01011637f92f6cd6f613cf176480b12a591480"} Nov 03 17:25:41.923903 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.923881 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs" Nov 03 17:25:41.924119 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:41.923918 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f81662bf2e92f4fc1ddca6dd01011637f92f6cd6f613cf176480b12a591480" Nov 03 17:25:42.585366 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:25:42.585330 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:25:49.249346 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.249306 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l"] Nov 03 17:25:49.249823 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.249747 5946 cpu_manager.go:414] "containerMap: removing container" podUID="1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" containerName="revoke-keys" Nov 03 17:25:49.249823 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.249763 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" containerName="revoke-keys" Nov 03 17:25:49.249952 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.249851 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" containerName="revoke-keys" Nov 03 17:25:49.253056 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.253034 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.309919 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.309881 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:25:49.359048 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.359001 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:25:49.359048 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.359042 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:25:49.359312 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.359084 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6jprr\"" Nov 03 17:25:49.360842 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.360821 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l"] Nov 03 17:25:49.383933 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.383903 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.384079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.383944 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhg5\" (UniqueName: \"kubernetes.io/projected/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-kube-api-access-srhg5\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.384079 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.383971 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.452446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.452388 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:49.484777 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.484749 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.484924 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.484789 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhg5\" (UniqueName: \"kubernetes.io/projected/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-kube-api-access-srhg5\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.484924 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.484811 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.485420 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.485399 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.487393 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.487368 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.546450 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.546378 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhg5\" (UniqueName: \"kubernetes.io/projected/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-kube-api-access-srhg5\") pod \"maas-api-revoke-sub-6b19338f-6q55l\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.569194 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.569165 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:49.643023 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.642992 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:49.944722 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.944626 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l"] Nov 03 17:25:49.947893 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:25:49.947862 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2f3c00_bf7c_4626_ba8f_a929d2da613a.slice/crio-cb5b99efbc00cca743572f899b393a122b48f5c27df2e3bfb1989a9c43565099 WatchSource:0}: Error finding container cb5b99efbc00cca743572f899b393a122b48f5c27df2e3bfb1989a9c43565099: Status 404 returned error can't find the container with id cb5b99efbc00cca743572f899b393a122b48f5c27df2e3bfb1989a9c43565099 Nov 03 17:25:49.963197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:49.963168 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" event={"ID":"5f2f3c00-bf7c-4626-ba8f-a929d2da613a","Type":"ContainerStarted","Data":"cb5b99efbc00cca743572f899b393a122b48f5c27df2e3bfb1989a9c43565099"} Nov 03 17:25:50.970913 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:50.970874 5946 generic.go:356] "Generic (PLEG): container finished" podID="5f2f3c00-bf7c-4626-ba8f-a929d2da613a" containerID="60fd7c6ac608c704d8db878ecfcbe52f29d47cdb1ddccb02644b2aa239de2a7f" exitCode=0 Nov 03 17:25:50.971377 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:50.970959 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" event={"ID":"5f2f3c00-bf7c-4626-ba8f-a929d2da613a","Type":"ContainerDied","Data":"60fd7c6ac608c704d8db878ecfcbe52f29d47cdb1ddccb02644b2aa239de2a7f"} Nov 03 17:25:52.113935 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.113910 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:52.310310 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.310275 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-kube-api-access-srhg5\" (UniqueName: \"kubernetes.io/projected/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-kube-api-access-srhg5\") pod \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " Nov 03 17:25:52.310492 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.310382 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-xks-ca\") pod \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " Nov 03 17:25:52.310492 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.310456 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-openshift-ca\") pod \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\" (UID: \"5f2f3c00-bf7c-4626-ba8f-a929d2da613a\") " Nov 03 17:25:52.310801 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.310777 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-openshift-ca" pod "5f2f3c00-bf7c-4626-ba8f-a929d2da613a" (UID: "5f2f3c00-bf7c-4626-ba8f-a929d2da613a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:52.312602 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.312576 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-xks-ca" pod "5f2f3c00-bf7c-4626-ba8f-a929d2da613a" (UID: "5f2f3c00-bf7c-4626-ba8f-a929d2da613a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:52.312698 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.312680 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-kube-api-access-srhg5" pod "5f2f3c00-bf7c-4626-ba8f-a929d2da613a" (UID: "5f2f3c00-bf7c-4626-ba8f-a929d2da613a"). InnerVolumeSpecName "kube-api-access-srhg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:52.412097 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.412065 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-xks-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:52.412097 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.412091 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-maas-api-openshift-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:52.412097 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.412102 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhg5\" (UniqueName: \"kubernetes.io/projected/5f2f3c00-bf7c-4626-ba8f-a929d2da613a-kube-api-access-srhg5\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:52.982552 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.982518 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" Nov 03 17:25:52.982743 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.982556 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l" event={"ID":"5f2f3c00-bf7c-4626-ba8f-a929d2da613a","Type":"ContainerDied","Data":"cb5b99efbc00cca743572f899b393a122b48f5c27df2e3bfb1989a9c43565099"} Nov 03 17:25:52.982743 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:25:52.982604 5946 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb5b99efbc00cca743572f899b393a122b48f5c27df2e3bfb1989a9c43565099" Nov 03 17:26:02.157571 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:02.157535 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v7f8v"] Nov 03 17:26:12.584725 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:26:12.584685 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:26:19.074372 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:19.074340 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:22.994216 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:22.994182 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-tzxp6_6fdb25a7-9065-46e7-b5ba-361ee9c1a10a/manager/0.log" Nov 03 17:26:23.926716 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:23.926687 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xz8cb_b4208b06-4937-4c72-bec3-b23d6c6dde43/server/0.log" Nov 03 17:26:24.404251 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:24.404221 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p2gwf_c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd/proxy/0.log" Nov 03 17:26:25.976675 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:25.976642 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-f2xrs_1d39c8bb-9e3d-49e6-acc7-43a20ce2502d/revoke-keys/0.log" Nov 03 17:26:26.210624 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:26.210596 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-6q55l_5f2f3c00-bf7c-4626-ba8f-a929d2da613a/revoke-keys/0.log" Nov 03 17:26:26.426544 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:26.426510 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-d9wtx_831a0d73-e900-424e-a8b3-cef70a821cf9/revoke-keys/0.log" Nov 03 17:26:28.823619 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:28.823592 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kdj6j_5a5cffea-f020-4efd-afab-ef98eecb3481/manager/0.log" Nov 03 17:26:29.762488 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:29.762451 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v7f8v_638519fb-1cb1-4d77-82b6-b29e1c8981f4/limitador/0.log" Nov 03 17:26:41.416561 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:41.416525 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbcdb5d6b-dzvh2_dfe8b267-f4d9-4984-80ca-fca687791b68/kube-auth-proxy/0.log" Nov 03 17:26:42.583640 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:26:42.583597 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:26:42.884755 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:42.884675 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-hcq7f_1d902919-9525-449c-9b32-c58e0fde1618/istio-proxy/0.log" Nov 03 17:26:43.143625 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:43.143547 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-kw9pt_f2070296-e232-4796-a766-50240d2a2286/istio-proxy/0.log" Nov 03 17:26:43.959969 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:43.959937 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-w8lvg_72b9663c-c665-4a5a-a2c7-c564bd2696b0/istio-proxy/0.log" Nov 03 17:26:44.449443 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:44.449412 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd_77e8e85e-5124-4a7e-85e5-7373fba14616/istio-proxy/0.log" Nov 03 17:26:49.775044 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:26:49.775012 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:10.692109 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:10.692082 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:27:10.711629 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:10.711598 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8r945_61daa065-4042-47db-add5-514189226920/kube-multus/0.log" Nov 03 17:27:12.585259 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:27:12.585194 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:27:38.657118 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:38.657085 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-tzxp6_6fdb25a7-9065-46e7-b5ba-361ee9c1a10a/manager/0.log" Nov 03 17:27:39.435838 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:39.435806 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xz8cb_b4208b06-4937-4c72-bec3-b23d6c6dde43/server/0.log" Nov 03 17:27:39.963628 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:39.963584 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p2gwf_c2e8b5ac-ddb5-42ed-a5f6-0e2b266441fd/proxy/0.log" Nov 03 17:27:41.168356 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:41.168322 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-f2xrs_1d39c8bb-9e3d-49e6-acc7-43a20ce2502d/revoke-keys/0.log" Nov 03 17:27:41.393631 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:41.393601 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-6q55l_5f2f3c00-bf7c-4626-ba8f-a929d2da613a/revoke-keys/0.log" Nov 03 17:27:41.603729 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:41.603704 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-d9wtx_831a0d73-e900-424e-a8b3-cef70a821cf9/revoke-keys/0.log" Nov 03 17:27:42.585391 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:27:42.585354 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:27:43.731990 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:43.731954 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kdj6j_5a5cffea-f020-4efd-afab-ef98eecb3481/manager/0.log" Nov 03 17:27:44.451288 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:44.451256 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v7f8v_638519fb-1cb1-4d77-82b6-b29e1c8981f4/limitador/0.log" Nov 03 17:27:45.860128 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:45.860097 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbcdb5d6b-dzvh2_dfe8b267-f4d9-4984-80ca-fca687791b68/kube-auth-proxy/0.log" Nov 03 17:27:47.442626 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:47.442596 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-hcq7f_1d902919-9525-449c-9b32-c58e0fde1618/istio-proxy/0.log" Nov 03 17:27:47.756361 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:47.756333 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-kw9pt_f2070296-e232-4796-a766-50240d2a2286/istio-proxy/0.log" Nov 03 17:27:48.618836 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:48.618807 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-w8lvg_72b9663c-c665-4a5a-a2c7-c564bd2696b0/istio-proxy/0.log" Nov 03 17:27:49.159650 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:49.159622 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-lkzrd_77e8e85e-5124-4a7e-85e5-7373fba14616/istio-proxy/0.log" Nov 03 17:27:50.159017 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:27:50.158984 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:12.584912 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:28:12.584873 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:28:13.460521 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:28:13.460488 5946 scope.go:122] "RemoveContainer" containerID="94359950a7bcbafce0f44fa9f814cc03291abac9352a4678528d26525112b6cf" Nov 03 17:28:42.584730 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:28:42.584695 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:28:50.545796 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:28:50.545751 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:52.231034 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:28:52.230988 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:52.422316 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:28:52.422277 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:03.640877 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:29:03.640844 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:12.584374 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:29:12.584332 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:29:42.584097 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:29:42.584059 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:29:50.884735 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:29:50.884680 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:00.187068 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.187033 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt"] Nov 03 17:30:00.187711 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.187655 5946 cpu_manager.go:414] "containerMap: removing container" podUID="5f2f3c00-bf7c-4626-ba8f-a929d2da613a" containerName="revoke-keys" Nov 03 17:30:00.187711 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.187676 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2f3c00-bf7c-4626-ba8f-a929d2da613a" containerName="revoke-keys" Nov 03 17:30:00.187845 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.187783 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2f3c00-bf7c-4626-ba8f-a929d2da613a" containerName="revoke-keys" Nov 03 17:30:00.190798 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.190777 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.246177 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.246147 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:30:00.259546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.259515 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76ph\" (UniqueName: \"kubernetes.io/projected/d7525d21-1015-472c-b213-115dacd9c328-kube-api-access-j76ph\") pod \"maas-api-key-cleanup-29895450-s4cdt\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.259701 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.259562 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7525d21-1015-472c-b213-115dacd9c328-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-s4cdt\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.296231 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.296168 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6jprr\"" Nov 03 17:30:00.296411 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.296345 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:30:00.298043 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.298019 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt"] Nov 03 17:30:00.361229 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.361167 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76ph\" (UniqueName: \"kubernetes.io/projected/d7525d21-1015-472c-b213-115dacd9c328-kube-api-access-j76ph\") pod \"maas-api-key-cleanup-29895450-s4cdt\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.361424 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.361260 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7525d21-1015-472c-b213-115dacd9c328-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-s4cdt\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.361892 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.361870 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7525d21-1015-472c-b213-115dacd9c328-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-s4cdt\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.422763 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.422730 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76ph\" (UniqueName: \"kubernetes.io/projected/d7525d21-1015-472c-b213-115dacd9c328-kube-api-access-j76ph\") pod \"maas-api-key-cleanup-29895450-s4cdt\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.503160 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.503132 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:00.882992 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:00.882907 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt"] Nov 03 17:30:00.886342 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:30:00.886305 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7525d21_1015_472c_b213_115dacd9c328.slice/crio-0b4464c7ee08fca6f2923be11c215474c23e088b7c62f1d3d280a13b1020454c WatchSource:0}: Error finding container 0b4464c7ee08fca6f2923be11c215474c23e088b7c62f1d3d280a13b1020454c: Status 404 returned error can't find the container with id 0b4464c7ee08fca6f2923be11c215474c23e088b7c62f1d3d280a13b1020454c Nov 03 17:30:01.282984 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:01.282951 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" event={"ID":"d7525d21-1015-472c-b213-115dacd9c328","Type":"ContainerStarted","Data":"db59dbbd9970bf26c6c3f7becef609f104d8eae246eff2da8d2a0ec288cd0b95"} Nov 03 17:30:01.283452 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:01.283005 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" event={"ID":"d7525d21-1015-472c-b213-115dacd9c328","Type":"ContainerStarted","Data":"0b4464c7ee08fca6f2923be11c215474c23e088b7c62f1d3d280a13b1020454c"} Nov 03 17:30:01.402632 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:01.402575 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" podStartSLOduration=1.4025583959999999 podStartE2EDuration="1.402558396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:01.401734349 +0000 UTC m=+4671.522787173" watchObservedRunningTime="2026-11-03 17:30:01.402558396 +0000 UTC m=+4671.523611221" Nov 03 17:30:02.289996 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:02.289962 5946 generic.go:356] "Generic (PLEG): container finished" podID="d7525d21-1015-472c-b213-115dacd9c328" containerID="db59dbbd9970bf26c6c3f7becef609f104d8eae246eff2da8d2a0ec288cd0b95" exitCode=60 Nov 03 17:30:02.290459 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:02.290050 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" event={"ID":"d7525d21-1015-472c-b213-115dacd9c328","Type":"ContainerDied","Data":"db59dbbd9970bf26c6c3f7becef609f104d8eae246eff2da8d2a0ec288cd0b95"} Nov 03 17:30:02.290459 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:02.290275 5946 scope.go:122] "RemoveContainer" containerID="db59dbbd9970bf26c6c3f7becef609f104d8eae246eff2da8d2a0ec288cd0b95" Nov 03 17:30:03.297791 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:03.297751 5946 generic.go:356] "Generic (PLEG): container finished" podID="d7525d21-1015-472c-b213-115dacd9c328" containerID="a609f1f2aa6f4982e91f82c31dc3eef2814aa8a3325c53575b168d247b4bbc16" exitCode=60 Nov 03 17:30:03.298257 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:03.297840 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" event={"ID":"d7525d21-1015-472c-b213-115dacd9c328","Type":"ContainerDied","Data":"a609f1f2aa6f4982e91f82c31dc3eef2814aa8a3325c53575b168d247b4bbc16"} Nov 03 17:30:03.298257 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:03.297924 5946 scope.go:122] "RemoveContainer" containerID="db59dbbd9970bf26c6c3f7becef609f104d8eae246eff2da8d2a0ec288cd0b95" Nov 03 17:30:03.298257 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:03.298062 5946 scope.go:122] "RemoveContainer" containerID="a609f1f2aa6f4982e91f82c31dc3eef2814aa8a3325c53575b168d247b4bbc16" Nov 03 17:30:03.298389 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:30:03.298354 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-s4cdt_odh-ai-gateway-infra(d7525d21-1015-472c-b213-115dacd9c328)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" podUID="d7525d21-1015-472c-b213-115dacd9c328" Nov 03 17:30:04.304791 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:04.304761 5946 scope.go:122] "RemoveContainer" containerID="a609f1f2aa6f4982e91f82c31dc3eef2814aa8a3325c53575b168d247b4bbc16" Nov 03 17:30:04.305186 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:30:04.304978 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-s4cdt_odh-ai-gateway-infra(d7525d21-1015-472c-b213-115dacd9c328)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" podUID="d7525d21-1015-472c-b213-115dacd9c328" Nov 03 17:30:09.891751 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:09.891715 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:10.092010 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:10.091978 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:12.584901 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:30:12.584864 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:30:16.003541 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:16.003514 5946 scope.go:122] "RemoveContainer" containerID="a609f1f2aa6f4982e91f82c31dc3eef2814aa8a3325c53575b168d247b4bbc16" Nov 03 17:30:16.366534 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:16.366499 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" event={"ID":"d7525d21-1015-472c-b213-115dacd9c328","Type":"ContainerStarted","Data":"6de04982727ba5af775379f0963c730e164a4bc937e57f370f7a58f023af4011"} Nov 03 17:30:17.373287 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:17.373253 5946 generic.go:356] "Generic (PLEG): container finished" podID="d7525d21-1015-472c-b213-115dacd9c328" containerID="6de04982727ba5af775379f0963c730e164a4bc937e57f370f7a58f023af4011" exitCode=60 Nov 03 17:30:17.373724 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:17.373321 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" event={"ID":"d7525d21-1015-472c-b213-115dacd9c328","Type":"ContainerDied","Data":"6de04982727ba5af775379f0963c730e164a4bc937e57f370f7a58f023af4011"} Nov 03 17:30:17.373724 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:17.373384 5946 scope.go:122] "RemoveContainer" containerID="a609f1f2aa6f4982e91f82c31dc3eef2814aa8a3325c53575b168d247b4bbc16" Nov 03 17:30:17.373724 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:17.373684 5946 scope.go:122] "RemoveContainer" containerID="6de04982727ba5af775379f0963c730e164a4bc937e57f370f7a58f023af4011" Nov 03 17:30:17.373948 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:30:17.373930 5946 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-s4cdt_odh-ai-gateway-infra(d7525d21-1015-472c-b213-115dacd9c328)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" podUID="d7525d21-1015-472c-b213-115dacd9c328" Nov 03 17:30:17.501858 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:17.501815 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt"] Nov 03 17:30:18.514761 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:18.514729 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:18.637195 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:18.637111 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7525d21-1015-472c-b213-115dacd9c328-kube-api-access-j76ph\" (UniqueName: \"kubernetes.io/projected/d7525d21-1015-472c-b213-115dacd9c328-kube-api-access-j76ph\") pod \"d7525d21-1015-472c-b213-115dacd9c328\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " Nov 03 17:30:18.637195 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:18.637170 5946 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7525d21-1015-472c-b213-115dacd9c328-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7525d21-1015-472c-b213-115dacd9c328-maas-api-service-ca\") pod \"d7525d21-1015-472c-b213-115dacd9c328\" (UID: \"d7525d21-1015-472c-b213-115dacd9c328\") " Nov 03 17:30:18.637942 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:18.637913 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7525d21-1015-472c-b213-115dacd9c328-maas-api-service-ca" pod "d7525d21-1015-472c-b213-115dacd9c328" (UID: "d7525d21-1015-472c-b213-115dacd9c328"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:18.638182 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:18.638159 5946 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7525d21-1015-472c-b213-115dacd9c328-maas-api-service-ca\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:18.640865 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:18.640791 5946 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7525d21-1015-472c-b213-115dacd9c328-kube-api-access-j76ph" pod "d7525d21-1015-472c-b213-115dacd9c328" (UID: "d7525d21-1015-472c-b213-115dacd9c328"). InnerVolumeSpecName "kube-api-access-j76ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:18.738952 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:18.738921 5946 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76ph\" (UniqueName: \"kubernetes.io/projected/d7525d21-1015-472c-b213-115dacd9c328-kube-api-access-j76ph\") on node \"ip-10-0-135-216.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:19.385977 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:19.385950 5946 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" Nov 03 17:30:19.386185 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:19.385947 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt" event={"ID":"d7525d21-1015-472c-b213-115dacd9c328","Type":"ContainerDied","Data":"0b4464c7ee08fca6f2923be11c215474c23e088b7c62f1d3d280a13b1020454c"} Nov 03 17:30:19.386185 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:19.386085 5946 scope.go:122] "RemoveContainer" containerID="6de04982727ba5af775379f0963c730e164a4bc937e57f370f7a58f023af4011" Nov 03 17:30:19.558149 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:19.558103 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt"] Nov 03 17:30:19.608421 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:19.608386 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-s4cdt"] Nov 03 17:30:19.814375 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:19.814335 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5dbjc_ab478348-e9ee-4b58-a398-422dd7e03f0b/global-pull-secret-syncer/0.log" Nov 03 17:30:20.003401 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:20.003371 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7525d21-1015-472c-b213-115dacd9c328" path="/var/lib/kubelet/pods/d7525d21-1015-472c-b213-115dacd9c328/volumes" Nov 03 17:30:20.067748 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:20.067672 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx"] Nov 03 17:30:20.075131 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:20.075108 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-d9wtx"] Nov 03 17:30:22.003264 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:22.003231 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831a0d73-e900-424e-a8b3-cef70a821cf9" path="/var/lib/kubelet/pods/831a0d73-e900-424e-a8b3-cef70a821cf9/volumes" Nov 03 17:30:23.003887 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:23.003854 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" Nov 03 17:30:24.484493 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:24.484465 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-216.us-west-2.compute.internal_4357665f1982cbd14d16f18df231ee45/haproxy/0.log" Nov 03 17:30:29.327197 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.327165 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kdj6j_5a5cffea-f020-4efd-afab-ef98eecb3481/manager/0.log" Nov 03 17:30:29.330896 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.330872 5946 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8"] Nov 03 17:30:29.331309 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331294 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.331360 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331310 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.331360 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331332 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.331360 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331337 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.331360 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331349 5946 cpu_manager.go:414] "containerMap: removing container" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.331360 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331353 5946 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.331551 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331439 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.331551 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.331452 5946 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7525d21-1015-472c-b213-115dacd9c328" containerName="cleanup" Nov 03 17:30:29.335593 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.335578 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.340414 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.340389 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-sys\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.340524 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.340433 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkz27\" (UniqueName: \"kubernetes.io/projected/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-kube-api-access-mkz27\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.340524 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.340497 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-lib-modules\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.340633 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.340563 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-proc\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.340633 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.340585 5946 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-podres\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.389589 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.389558 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwpzh\"/\"kube-root-ca.crt\"" Nov 03 17:30:29.389733 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.389595 5946 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwpzh\"/\"openshift-service-ca.crt\"" Nov 03 17:30:29.441126 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441097 5946 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mwpzh\"/\"default-dockercfg-c459v\"" Nov 03 17:30:29.441387 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441362 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-sys\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441447 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441402 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkz27\" (UniqueName: \"kubernetes.io/projected/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-kube-api-access-mkz27\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441447 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441427 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-lib-modules\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441482 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-sys\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441546 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441535 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-lib-modules\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441631 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441587 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-proc\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441631 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441620 5946 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-podres\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441711 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441652 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-proc\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.441711 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.441700 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-podres\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.442726 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.442700 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8"] Nov 03 17:30:29.502102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.502072 5946 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkz27\" (UniqueName: \"kubernetes.io/projected/4bdc3ec4-257b-469a-aa87-fd90a125aeaf-kube-api-access-mkz27\") pod \"perf-node-gather-daemonset-kf9t8\" (UID: \"4bdc3ec4-257b-469a-aa87-fd90a125aeaf\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:29.647406 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:29.647319 5946 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:30.021247 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:30.021222 5946 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8"] Nov 03 17:30:30.027271 ip-10-0-135-216 kubenswrapper[5946]: W1103 17:30:30.027233 5946 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bdc3ec4_257b_469a_aa87_fd90a125aeaf.slice/crio-053ce34336364932e7a2ba9a63f046a9656dcdda777755f19616d35580ba0620 WatchSource:0}: Error finding container 053ce34336364932e7a2ba9a63f046a9656dcdda777755f19616d35580ba0620: Status 404 returned error can't find the container with id 053ce34336364932e7a2ba9a63f046a9656dcdda777755f19616d35580ba0620 Nov 03 17:30:30.443875 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:30.443793 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" event={"ID":"4bdc3ec4-257b-469a-aa87-fd90a125aeaf","Type":"ContainerStarted","Data":"94bcfbf7addfb8eabaafd3b8cc875024f9bbc55588d03ee69a4ca64b9b57ec0a"} Nov 03 17:30:30.443875 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:30.443843 5946 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" event={"ID":"4bdc3ec4-257b-469a-aa87-fd90a125aeaf","Type":"ContainerStarted","Data":"053ce34336364932e7a2ba9a63f046a9656dcdda777755f19616d35580ba0620"} Nov 03 17:30:30.444335 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:30.443920 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:30.566391 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:30.566334 5946 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" podStartSLOduration=1.56631887 podStartE2EDuration="1.56631887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:30.565078164 +0000 UTC m=+4700.686130986" watchObservedRunningTime="2026-11-03 17:30:30.56631887 +0000 UTC m=+4700.687371720" Nov 03 17:30:30.653871 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:30.653837 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v7f8v_638519fb-1cb1-4d77-82b6-b29e1c8981f4/limitador/0.log" Nov 03 17:30:36.459020 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:36.458989 5946 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-kf9t8" Nov 03 17:30:37.348297 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:37.348264 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-wfsql_61a83e02-ce4d-4df6-bc29-368ec906215f/kube-state-metrics/0.log" Nov 03 17:30:37.816995 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:37.816959 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-wfsql_61a83e02-ce4d-4df6-bc29-368ec906215f/kube-rbac-proxy-main/0.log" Nov 03 17:30:38.251804 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:38.251753 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-wfsql_61a83e02-ce4d-4df6-bc29-368ec906215f/kube-rbac-proxy-self/0.log" Nov 03 17:30:38.691727 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:38.691652 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d7dfbc995-gq2ct_5a8ec6fc-a61e-4d85-b9c8-f594647b0a96/metrics-server/0.log" Nov 03 17:30:39.131353 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:39.131324 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-nw79m_cedb2971-5861-4832-bda4-242c206a0822/monitoring-plugin/0.log" Nov 03 17:30:40.967074 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:40.967041 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfllm_234cc11d-353c-4f0c-ab3b-4616ddd0b26d/node-exporter/0.log" Nov 03 17:30:41.418466 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:41.418441 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfllm_234cc11d-353c-4f0c-ab3b-4616ddd0b26d/kube-rbac-proxy/0.log" Nov 03 17:30:41.856446 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:41.856418 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfllm_234cc11d-353c-4f0c-ab3b-4616ddd0b26d/init-textfile/0.log" Nov 03 17:30:42.060706 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:42.060671 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs"] Nov 03 17:30:42.111473 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:42.111395 5946 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f2xrs"] Nov 03 17:30:42.585338 ip-10-0-135-216 kubenswrapper[5946]: E1103 17:30:42.585303 5946 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbst4" containerName="extractor" Nov 03 17:30:43.619764 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:43.619736 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bcvnw_3833015d-9c9e-49d7-8630-a4b118d57ab6/kube-rbac-proxy-main/0.log" Nov 03 17:30:44.003053 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:44.003022 5946 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d39c8bb-9e3d-49e6-acc7-43a20ce2502d" path="/var/lib/kubelet/pods/1d39c8bb-9e3d-49e6-acc7-43a20ce2502d/volumes" Nov 03 17:30:44.055916 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:44.055889 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bcvnw_3833015d-9c9e-49d7-8630-a4b118d57ab6/kube-rbac-proxy-self/0.log" Nov 03 17:30:44.492256 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:44.492229 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bcvnw_3833015d-9c9e-49d7-8630-a4b118d57ab6/openshift-state-metrics/0.log" Nov 03 17:30:44.939553 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:44.939456 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e29f1414-146f-4bb2-a62c-a29b6f54ff75/prometheus/0.log" Nov 03 17:30:45.379443 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:45.379414 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e29f1414-146f-4bb2-a62c-a29b6f54ff75/config-reloader/0.log" Nov 03 17:30:45.816163 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:45.816133 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e29f1414-146f-4bb2-a62c-a29b6f54ff75/thanos-sidecar/0.log" Nov 03 17:30:46.252017 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:46.251972 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e29f1414-146f-4bb2-a62c-a29b6f54ff75/kube-rbac-proxy-web/0.log" Nov 03 17:30:46.690936 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:46.690854 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e29f1414-146f-4bb2-a62c-a29b6f54ff75/kube-rbac-proxy/0.log" Nov 03 17:30:47.130280 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:47.130174 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e29f1414-146f-4bb2-a62c-a29b6f54ff75/kube-rbac-proxy-thanos/0.log" Nov 03 17:30:47.566791 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:47.566765 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e29f1414-146f-4bb2-a62c-a29b6f54ff75/init-config-reloader/0.log" Nov 03 17:30:50.645804 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:50.645741 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b684bf768-ghn5f_5e9344a7-9120-40ef-8671-5bc1fd4de206/thanos-query/0.log" Nov 03 17:30:51.079310 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:51.079280 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b684bf768-ghn5f_5e9344a7-9120-40ef-8671-5bc1fd4de206/kube-rbac-proxy-web/0.log" Nov 03 17:30:51.172221 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:51.172176 5946 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm72k_9189a7df-aae8-4728-be2b-569f2cffebc3/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:51.513381 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:51.513352 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b684bf768-ghn5f_5e9344a7-9120-40ef-8671-5bc1fd4de206/kube-rbac-proxy/0.log" Nov 03 17:30:51.949102 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:51.949022 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b684bf768-ghn5f_5e9344a7-9120-40ef-8671-5bc1fd4de206/prom-label-proxy/0.log" Nov 03 17:30:52.425220 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:52.425178 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b684bf768-ghn5f_5e9344a7-9120-40ef-8671-5bc1fd4de206/kube-rbac-proxy-rules/0.log" Nov 03 17:30:52.541333 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:52.541304 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbcdb5d6b-dzvh2_dfe8b267-f4d9-4984-80ca-fca687791b68/kube-auth-proxy/0.log" Nov 03 17:30:52.862537 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:52.862505 5946 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b684bf768-ghn5f_5e9344a7-9120-40ef-8671-5bc1fd4de206/kube-rbac-proxy-metrics/0.log" Nov 03 17:30:53.062917 ip-10-0-135-216 kubenswrapper[5946]: I1103 17:30:53.062885 5946 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6q55l"]